Image processing methods, apparatuses, electronic devices, and storage media

Abstract

This disclosure relates to the field of computer vision technology, specifically providing an image processing method, apparatus, electronic device, and storage medium. An image processing method includes: extracting features from a video stream sequence to obtain an image feature sequence and a difference feature sequence for inter-frame images; performing fusion processing based on the image feature sequence and the difference feature sequence to obtain a first fused feature sequence; processing the first fused feature sequence according to the correlation weights of the inter-frame images to obtain a second fused feature sequence; and determining at least one keyframe image from the video stream sequence based on the second fused feature sequence. In this embodiment, the extracted keyframe image simultaneously considers inter-frame correlation and difference, improving the accuracy of keyframe extraction.

Description

Technical Field

[0001] This disclosure relates to the field of computer vision technology, specifically to an image processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] Video keyframe extraction refers to selecting one or more of the most representative video frames from a video stream. It has wide applications in various scenarios. For example, keyframe extraction from traffic videos and exhibition videos can quickly locate suspicious events and shorten video playback time; similarly, keyframe extraction from entertainment videos and news videos can quickly retrieve video clips of interest to users.

[0003] Among related technologies, keyframe extraction from video streams can be achieved based on deep learning networks (DNNs). However, the current performance of DNN-based video keyframe extraction is not ideal. Summary of the Invention

[0004] To improve the extraction effect of video keyframes, this disclosure provides an image processing method, apparatus, electronic device, and storage medium.

[0005] In a first aspect, embodiments of this disclosure provide an image processing method, including:

[0006] Feature extraction is performed on the video stream sequence to obtain the image feature sequence of the video stream sequence and the difference feature sequence for inter-frame images;

[0007] A first fused feature sequence is obtained by fusing the image feature sequence and the difference feature sequence.

[0008] The first fused feature sequence is processed based on the correlation weights of the inter-frame images to obtain the second fused feature sequence;

[0009] Based on the second fusion feature sequence, at least one keyframe image is determined from the video stream sequence.

[0010] In some implementations, feature extraction is performed on the video stream sequence to obtain an image feature sequence of the video stream sequence, including:

[0011] Feature extraction is performed on the video stream sequence to obtain the image features of each frame in the video stream sequence;

[0012] Based on the image features of each frame, the image feature sequence of the video stream sequence is determined.

[0013] In some implementations, feature extraction is performed on the video stream sequence to obtain a sequence of differential features for inter-frame images, including:

[0014] For any frame of the video stream sequence, forward difference features are determined based on the image features of the image and the image features of the previous frame.

[0015] Based on the image features of the image and the image features of the next frame, the backward difference features are determined.

[0016] Based on the forward difference features and the backward difference features of all frame images in the video stream sequence, the difference feature sequence of the video stream sequence is determined.

[0017] In some embodiments, the step of fusing the image feature sequence and the difference feature sequence to obtain a first fused feature sequence includes:

[0018] The target weights of the differential feature sequence are determined based on the rate of difference change between the frames in the differential feature sequence; the rate of difference change is determined based on the differential features of each frame.

[0019] The first fused feature sequence is obtained by fusing the image feature sequence, the target weight, and the difference feature sequence.

[0020] In some implementations, processing the first fused feature sequence based on the correlation weights of the inter-frame images to obtain the second fused feature sequence includes:

[0021] The correlation weights of the inter-frame images are determined based on the self-attention mechanism.

[0022] The first fused feature sequence is processed according to the correlation weights to obtain the second fused feature sequence.

[0023] In some implementations, determining at least one keyframe image from the video stream sequence based on the second fused feature sequence includes:

[0024] Based on the second fused feature sequence, the importance score of each frame image is predicted;

[0025] Image detection is performed on the video stream sequence to determine at least one scene switching node in the video stream sequence;

[0026] The video stream sequence is segmented based on the at least one scene switching node to obtain at least two video sequence segments;

[0027] The keyframe image is determined based on the length of each video sequence segment and the importance score of each frame in the video sequence segment.

[0028] In some implementations, the step of extracting features from the video stream sequence to obtain an image feature sequence of the video stream sequence and a difference feature sequence for inter-frame images includes:

[0029] The video stream sequence is input into a pre-trained feature extraction network to obtain the image feature sequence output by the feature extraction network;

[0030] The image feature sequence is input into a pre-trained temporal difference network to obtain the difference feature sequence output by the temporal difference network.

[0031] The process of fusing the image feature sequence and the difference feature sequence to obtain the first fused feature sequence includes:

[0032] The temporal difference network performs fusion processing based on the image feature sequence and the difference feature sequence, and outputs the first fused feature sequence;

[0033] The step of processing the first fused feature sequence according to the correlation weights of the inter-frame images to obtain the second fused feature sequence includes:

[0034] The first fused feature sequence is input into a pre-trained self-attention network, so that the self-attention network processes the first fused feature sequence according to the determined relevance weights and outputs the second fused feature sequence.

[0035] Determining at least one keyframe image from the video stream sequence based on the second fused feature sequence includes:

[0036] The second fused feature sequence is input into a pre-trained regression network to obtain the prediction result output by the regression network, which represents the prediction result of the keyframe image.

[0037] In some implementations, the method further includes a training process for at least one of the feature extraction network, the temporal difference network, the self-attention network, and the regression network, the training process including:

[0038] Acquire sample data, which includes a video stream sample sequence, a score label for each frame in the video stream sample sequence, and a keyframe label for each frame; the video stream sample sequence includes at least two keyframe segments;

[0039] Based on the keyframe labels, a preset number of non-keyframe images adjacent to each keyframe segment are determined as target frame segments.

[0040] For any keyframe segment, based on the image features of the keyframe segment and the image features of each target frame segment, a first similarity feature between the keyframe segment and the corresponding target frame segment, and a second similarity feature between the keyframe segment and each non-corresponding target frame segment are determined.

[0041] The contrast loss between frames is determined by comparing the ratio of the first similarity feature to the second similarity feature corresponding to each keyframe segment.

[0042] Based on the contrastive loss, at least one of the feature extraction network, the temporal difference network, the self-attention network, and the regression network to be trained is adjusted in terms of parameters until the convergence condition is met.

[0043] In some implementations, the training process further includes:

[0044] The video stream sample sequence is input into the feature extraction network to be trained to obtain the image feature sequence output by the feature extraction network;

[0045] The image feature sequence is input into the temporal difference network to be trained to obtain the first fused feature sequence output by the temporal difference network;

[0046] The first fused feature sequence is input into the self-attention network to be trained to obtain the second fused feature sequence output by the self-attention network;

[0047] The second fused feature sequence is input into the regression network to be trained to obtain the prediction result output by the regression network; the prediction result represents the importance score of each frame image in the video stream sample sequence;

[0048] Based on the prediction results and the differences between the score labels, at least one of the feature extraction network, the temporal difference network, the self-attention network, and the regression network to be trained is adjusted in terms of parameters until the convergence condition is met.

[0049] In a second aspect, embodiments of this disclosure provide an image processing apparatus, including:

[0050] The feature extraction module is configured to extract features from the video stream sequence to obtain an image feature sequence of the video stream sequence and a difference feature sequence for inter-frame images.

[0051] The differential fusion module is configured to perform fusion processing based on the image feature sequence and the differential feature sequence to obtain a first fused feature sequence;

[0052] The self-attention module is configured to process the first fused feature sequence based on the correlation weights of the inter-frame images to obtain the second fused feature sequence;

[0053] The keyframe determination module is configured to determine at least one keyframe image from the video stream sequence based on the second fused feature sequence.

[0054] In some implementations, the feature extraction module is configured to:

[0055] Feature extraction is performed on the video stream sequence to obtain the image features of each frame in the video stream sequence;

[0056] Based on the image features of each frame, the image feature sequence of the video stream sequence is determined.

[0057] In some implementations, the feature extraction module is configured to:

[0058] For any frame of the video stream sequence, forward difference features are determined based on the image features of the image and the image features of the previous frame.

[0059] Based on the image features of the image and the image features of the next frame, the backward difference features are determined.

[0060] Based on the forward difference features and the backward difference features of all frame images in the video stream sequence, the difference feature sequence of the video stream sequence is determined.

[0061] In some implementations, the differential fusion module is configured to:

[0062] The target weights of the differential feature sequence are determined based on the rate of difference change between the frames in the differential feature sequence; the rate of difference change is determined based on the differential features of each frame.

[0063] The first fused feature sequence is obtained by fusing the image feature sequence, the target weight, and the difference feature sequence.

[0064] In some implementations, the self-attention module is configured to:

[0065] The correlation weights of the inter-frame images are determined based on the self-attention mechanism.

[0066] The first fused feature sequence is processed according to the correlation weights to obtain the second fused feature sequence.

[0067] In some implementations, the keyframe determination module is configured to:

[0068] Based on the second fused feature sequence, the importance score of each frame image is predicted;

[0069] Image detection is performed on the video stream sequence to determine at least one scene switching node in the video stream sequence;

[0070] The video stream sequence is segmented based on the at least one scene switching node to obtain at least two video sequence segments;

[0071] The keyframe image is determined based on the length of each video sequence segment and the importance score of each frame in the video sequence segment.

[0072] In some implementations, the feature extraction module is configured to: input the video stream sequence into a pre-trained feature extraction network to obtain the image feature sequence output by the feature extraction network; and input the image feature sequence into a pre-trained temporal difference network to obtain the difference feature sequence output by the temporal difference network.

[0073] The differential fusion module is configured such that the temporal differential network performs fusion processing based on the image feature sequence and the difference feature sequence, and outputs the first fused feature sequence;

[0074] The self-attention module is configured to: input the first fused feature sequence into a pre-trained self-attention network, so that the self-attention network processes the first fused feature sequence according to the determined relevance weights and outputs the second fused feature sequence;

[0075] The keyframe determination module is configured to: input the second fused feature sequence into a pre-trained regression network to obtain the prediction result output by the regression network, wherein the prediction result represents the prediction result of the keyframe image.

[0076] In some embodiments, the apparatus further includes a training module configured to:

[0077] Acquire sample data, which includes a video stream sample sequence, a score label for each frame in the video stream sample sequence, and a keyframe label for each frame; the video stream sample sequence includes at least two keyframe segments;

[0078] Based on the keyframe labels, a preset number of non-keyframe images adjacent to each keyframe segment are determined as target frame segments.

[0079] For any keyframe segment, based on the image features of the keyframe segment and the image features of each target frame segment, a first similarity feature between the keyframe segment and the corresponding target frame segment, and a second similarity feature between the keyframe segment and each non-corresponding target frame segment are determined.

[0080] The contrast loss between frames is determined by comparing the ratio of the first similarity feature to the second similarity feature corresponding to each keyframe segment.

[0081] Based on the contrastive loss, at least one of the feature extraction network, the temporal difference network, the self-attention network, and the regression network to be trained is adjusted in terms of parameters until the convergence condition is met.

[0082] In some implementations, the training module is configured to:

[0083] The video stream sample sequence is input into the feature extraction network to be trained to obtain the image feature sequence output by the feature extraction network;

[0084] The image feature sequence is input into the temporal difference network to be trained to obtain the first fused feature sequence output by the temporal difference network;

[0085] The first fused feature sequence is input into the self-attention network to be trained to obtain the second fused feature sequence output by the self-attention network;

[0086] The second fused feature sequence is input into the regression network to be trained to obtain the prediction result output by the regression network; the prediction result represents the importance score of each frame image in the video stream sample sequence;

[0087] Based on the prediction results and the differences between the score labels, at least one of the feature extraction network, the temporal difference network, the self-attention network, and the regression network to be trained is adjusted in terms of parameters until the convergence condition is met.

[0088] Thirdly, embodiments of this disclosure provide an electronic device, including:

[0089] Processor; and

[0090] The memory stores computer instructions that can be read by the processor, and when the computer instructions are read, the processor executes the method according to any embodiment of the first aspect.

[0091] Fourthly, embodiments of this disclosure provide a storage medium for storing computer-readable instructions for causing a computer to perform the method according to any embodiment of the first aspect.

[0092] The image processing method of this disclosure includes extracting features from a video stream sequence to obtain an image feature sequence and a difference feature sequence; fusing the image feature sequence and the difference feature sequence to obtain a first fused feature sequence; processing the first fused feature sequence according to the correlation weights of inter-frame images to obtain a second fused feature sequence; and determining at least one keyframe image from the video stream sequence based on the second fused feature sequence. In this disclosure, by combining the correlation of inter-frame images with the differences between them, the extracted keyframe image takes into account both inter-frame correlation and differences. Especially for video stream sequences with minimal content changes, this method can effectively reduce redundant frames in the extracted video keyframes and improve the accuracy of keyframe extraction. Attached Figure Description

[0093] To more clearly illustrate the technical solutions in the specific embodiments of this disclosure or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0094] Figure 1 This is a flowchart of an image processing method according to some embodiments of the present disclosure.

[0095] Figure 2 This is a structural diagram of an image detection network according to some embodiments of this disclosure.

[0096] Figure 3 This is a flowchart of an image processing method according to some embodiments of the present disclosure.

[0097] Figure 4 This is a flowchart of an image processing method according to some embodiments of the present disclosure.

[0098] Figure 5 This is a flowchart of an image processing method according to some embodiments of the present disclosure.

[0099] Figure 6 This is a flowchart of an image processing method according to some embodiments of the present disclosure.

[0100] Figure 7 This is a flowchart of an image processing method according to some embodiments of the present disclosure.

[0101] Figure 8 This is a flowchart of an image processing method according to some embodiments of the present disclosure.

[0102] Figure 9This is a flowchart of an image processing method according to some embodiments of the present disclosure.

[0103] Figure 10 This is a structural block diagram of an image processing apparatus according to some embodiments of the present disclosure.

[0104] Figure 11 This is a structural block diagram of an electronic device according to some embodiments of the present disclosure. Detailed Implementation

[0105] The technical solutions of this disclosure will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without inventive effort are within the scope of protection of this disclosure. Furthermore, the technical features involved in the different embodiments of this disclosure described below can be combined with each other as long as they do not conflict with each other.

[0106] Video keyframe extraction refers to selecting one or more of the most representative video frames from a video stream sequence.

[0107] For example, in a scenario where the video stream sequence is a surveillance video, keyframe extraction can quickly locate suspicious events within the video stream sequence, and the frames corresponding to these suspicious events can be used as keyframes. This significantly reduces playback time and improves the efficiency of surveillance video retrieval during playback.

[0108] For example, in another scenario, the video stream sequence is an entertainment video. Keyframes can be customized in advance according to user needs. By using video keyframe extraction, video clips of people of interest to the user can be extracted from a longer video stream sequence.

[0109] For example, in another scenario, the video stream sequence is a conference video. By using video keyframe extraction, representative video frames can be extracted, thus providing a more comprehensive and accurate summary of the main content of the entire conference video.

[0110] As demonstrated by the examples above, video keyframe extraction has wide applications in various scenarios. Among related technologies, keyframe extraction from video stream sequences can be achieved based on deep learning networks (DNNs). For example, neural network models can be constructed by introducing recurrent neural networks (RNNs), graph neural networks (GNNs), and attention mechanisms, and then these neural network models can be used to extract keyframes from video stream sequences.

[0111] However, in related technical solutions, neural networks built using RNNs, GNNs, and attention mechanisms only focus on the temporal dependencies between video frames. This means the network can only learn the correlations between video frames, losing the differences in video frame content. However, the inventors in this case, through comparative research, discovered that for most keyframe extraction tasks, it is necessary to compare the similarities and differences in video frame content. Therefore, while extracting the correlations, preserving the differences between video frames is crucial. In related technical solutions, because the neural network model loses the differences between video frames, many redundant frames are extracted, especially for videos with minimal content changes. This deficiency is more severe, leading to poor keyframe extraction results.

[0112] In view of the deficiencies existing in the above-mentioned related technologies, the present disclosure provides an image processing method, apparatus, electronic device and storage medium, which aims to improve the effect of video keyframe extraction.

[0113] In a first aspect, embodiments of this disclosure provide an image processing method that can be applied to an electronic device and executed by the processor of the electronic device. The electronic device described in embodiments of this disclosure can be any suitable type of device, such as user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, vehicle-mounted device, wearable device, etc., and this disclosure does not impose any limitations thereon.

[0114] like Figure 1 As shown, in some embodiments, the image processing method of this disclosure includes:

[0115] S110. Extract features from the video stream sequence to obtain the image feature sequence of the video stream sequence and the difference feature sequence for inter-frame images.

[0116] In this embodiment of the disclosure, the video stream sequence represents a video stream comprising multiple consecutive frames of images, wherein each frame of the video stream sequence has temporal continuity with the previous frame and the next frame.

[0117] For example, in some embodiments, the video stream sequence may be a video stream data captured by a surveillance camera; in other embodiments, the video stream sequence may be a video stream data generated by an application; etc., this disclosure does not limit it.

[0118] Image detection technology can be used to extract features from each frame of a video stream sequence, obtaining image features of the video frame image. Image features can include semantic features, texture features, etc. It can be understood that through feature extraction, the image features corresponding to each frame image can be obtained, and the sequence composed of the image features of all frames in the video stream sequence is the image feature sequence described in the embodiments of this disclosure.

[0119] In some implementations, feature extraction of video stream sequences can be achieved using a pre-trained feature extraction network, which can be, for example, a network structure based on convolutional neural networks (CNNs) such as GoogleNet or VGG (Visual Geometry Group).

[0120] In this embodiment of the disclosure, after obtaining the image feature sequence of the video stream sequence, it is also necessary to calculate the difference feature sequence between consecutive frame images based on the image feature sequence.

[0121] The difference feature sequence is used to characterize the differences between frame images. It is understood that for any two adjacent frame images, due to changes in image content, there are differences in the image features corresponding to the adjacent frame images. In the embodiments of this disclosure, the difference feature sequence of the video stream sequence is determined based on the feature differences between adjacent frame images, and the difference features are fused into the original image feature sequence, thereby fully considering the differences between images in the video stream frames. This will be described in detail below.

[0122] In some embodiments, the difference feature sequence may include forward difference features and / or backward difference features of the inter-frame images of the video stream. Forward difference features represent the difference in image features between a given frame and the previous frame, while backward difference features represent the difference in image features between a given frame and the next frame. In embodiments of this disclosure, only the forward difference features of the video stream sequence may be used as the difference feature sequence, or only the backward difference features of the video stream sequence may be used as the difference feature sequence, or both forward and backward difference features may be used simultaneously. This will be described in detail below.

[0123] S120. Based on the image feature sequence and the difference feature sequence, perform fusion processing to obtain the first fused feature sequence.

[0124] As mentioned above, image feature sequences represent the original image features of a video stream sequence, while difference feature sequences represent the inter-frame differences in a video stream sequence. Therefore, fusing the difference feature sequences with the image feature sequences can fully incorporate the differences in video stream content in the first fused feature sequence, which is beneficial to the accuracy of subsequent keyframe extraction.

[0125] In some implementations, the differential changes between frames of a video stream can be captured using a Long Short-Term Memory (LSTM) network, and corresponding fusion parameters can be determined based on these differential changes. The fusion parameters are then used to perform feature fusion processing on the image feature sequence and the differential feature sequence to obtain a first fused feature sequence that incorporates the inter-frame differences. This will be described in the following implementations of this disclosure, and will not be detailed here.

[0126] S130. The first fusion feature sequence is processed according to the correlation weights of the inter-frame images to obtain the second fusion feature sequence.

[0127] As can be seen from the foregoing, the differences between the frames are fully integrated in the first fusion feature sequence obtained above. For the task of extracting keyframes from videos, it is also necessary to fully consider the correlation between the frames of the video stream sequence.

[0128] In some embodiments of this disclosure, the relevance weights of inter-frame images in a video stream sequence can be determined based on a self-attention mechanism. It is understood that the attention mechanism allows the network to better focus on the relevance of inter-frame images in the video stream. By processing the first fused feature sequence based on the relevance weights, the relevance of inter-frame images can be fused into the first fused feature sequence, resulting in a second fused feature sequence that includes both inter-frame image relevance and inter-frame image differences.

[0129] The process of processing the first fused feature sequence according to the correlation weight is described in the following embodiments of this disclosure, and will not be detailed here.

[0130] S140. Determine at least one keyframe image from the video stream sequence based on the second fusion feature sequence.

[0131] As can be seen from the foregoing, the second fused feature sequence includes both the correlation feature representation and the difference feature representation of the inter-frame images. Therefore, when making key predictions on the frame images in the video stream sequence based on the second fused feature sequence, the correlation and difference between the inter-frame images can be fully considered, reducing redundant frames in the key frame images and improving the accuracy of video key frame extraction.

[0132] In some implementations, a second fused feature sequence can be input into a pre-trained regression network, thereby enabling the regression network to predict and output an importance score for each frame in the video stream sequence. Then, one or more keyframe images in the video stream sequence are determined based on the importance scores of each frame. This will be described in the following embodiments of the present disclosure, and will not be detailed here.

[0133] As can be seen from the above, in this embodiment of the present disclosure, based on the correlation between images between video stream frames and combined with the differences between images between frames, the extracted keyframe images take into account both the correlation and differences between frames. Especially for video stream sequences with small content changes, it can effectively reduce redundant frames in the extracted video keyframes and improve the accuracy of keyframe extraction.

[0134] In some embodiments, the image processing method of this disclosure can be implemented using a DNN-based image detection network. Figure 2 The structure of the image detection network in some embodiments of this disclosure is shown below, in conjunction with... Figure 2 Please provide an explanation.

[0135] like Figure 2 As shown, in some embodiments, the image detection network of this disclosure includes a feature extraction network 100, a temporal difference network 200, a self-attention network 300, and a regression network 400.

[0136] The feature extraction network 100 takes a video stream sequence as input and outputs an image feature sequence corresponding to the video stream sequence. In some implementations, the feature extraction network 100 may employ neural network architectures such as GoogleNet or VGG. The feature extraction network 100 may extract features from each frame of the video stream sequence through, for example, convolutional processing, thereby obtaining the image features corresponding to each frame. The image features of all frames constitute the image feature sequence of the video stream sequence.

[0137] The input of the temporal difference network 200 is the image feature sequence output by the feature extraction network 100. The temporal difference network 200 can obtain the difference feature sequence by processing the image feature sequence, and then fuse the difference feature sequence with the original image feature sequence, so that the original image feature sequence can fuse the inter-frame difference features to obtain the first fused feature sequence.

[0138] The input to the self-attention network 300 is the first fused feature sequence output by the temporal difference network 200. The function of the self-attention network 300 is to perform correlation processing on the first fused feature sequence based on the self-attention mechanism, so that the temporal correlation between the inter-frame images is further fused in the first fused feature sequence to obtain the second fused feature sequence.

[0139] The input to the regression network 400 is the second fused feature sequence output by the self-attention network 300. The regression network 400 is used to predict the importance score of each frame image based on the first fused feature sequence. Then, the key frame images can be determined from the video stream sequence based on the importance score output by the regression network 400 through post-processing logic.

[0140] exist Figure 2 Based on the image detection network shown, the image processing method of this disclosure embodiment will be further described below.

[0141] like Figure 3 As shown, in some embodiments, the image processing method of this disclosure, which extracts features from a video stream to obtain an image feature sequence, may include:

[0142] S310. Extract features from the video stream sequence to obtain the image features of each frame in the video stream sequence.

[0143] S320. Determine the image feature sequence of the video stream sequence based on the image features of each frame.

[0144] like Figure 2 As shown, a video stream sequence can be input into a pre-trained feature extraction network 100, so that the feature extraction network 100 can extract features from each frame of the video stream sequence. The image features extracted by the feature extraction network 100 may include, for example, semantic features, texture features, etc., so that the image features corresponding to each frame of the image can be obtained.

[0145] It can be understood that the frames in a video stream sequence are continuous in time, and the image features corresponding to each frame extracted also have continuity in time. That is, based on the image features of all frames, the image feature sequence corresponding to the video stream sequence can be determined.

[0146] After obtaining the image feature sequence of the video stream sequence, the difference feature sequence that identifies the differences between frames can be determined based on the image feature sequence. The following section combines... Figure 4 The implementation method is described below.

[0147] like Figure 4 As shown, in some embodiments, the image processing method of this disclosure, in the process of determining the differential feature sequence, includes:

[0148] S410. For any frame of an image in a video stream sequence, determine the forward difference features based on the image features of the image and the image features of the previous frame.

[0149] S420. Based on the image features of the image and the image features of the next frame, determine the backward difference features.

[0150] S430. Based on the forward and backward difference features of all frame images in the video stream sequence, determine the difference feature sequence of the video stream sequence.

[0151] As mentioned above, the image feature sequence includes the image features corresponding to each frame. For example, in one example, the video stream sequence is represented as v = [v1, v2, ..., vn], consisting of n video frames. Figure 3 The implementation method determines that the image feature sequence corresponding to the video stream sequence is represented as f = [f1, f2, ..., fn], which contains a total of n image features.

[0152] In this embodiment of the disclosure, the difference feature sequence includes a forward difference feature sequence and a backward difference feature sequence.

[0153] The forward difference feature sequence represents the difference in image features between each frame and the previous frame. Taking one frame as an example, we define this frame as the "current frame image". In one example, the current frame image is v. m The corresponding image feature is f m The current frame image v m The previous frame image is v m-1 The corresponding image feature is f m-1 Therefore, the forward difference feature corresponding to the current frame image is f. m -f m-1 The forward difference features between every two adjacent frames are calculated sequentially to obtain the forward difference feature sequence corresponding to the entire video stream sequence, denoted as d1.

[0154] The backward difference feature sequence represents the difference in image features between each frame and the next frame. Taking one frame as an example, we define this frame as the "current frame image". In one example, the current frame image is v. m The corresponding image feature is f m The current frame image v m The next frame image is v m+1 The corresponding image feature is f m+1 Therefore, the backward difference feature corresponding to the current frame image is f. m -f m+1 The backward difference features between every two adjacent frames are calculated sequentially to obtain the backward difference feature sequence corresponding to the entire video stream sequence, denoted as d2.

[0155] In this embodiment of the disclosure, the forward difference feature sequence and the backward difference feature sequence together constitute the difference feature sequence, that is, the difference feature sequence described in this disclosure includes both the forward difference feature sequence and the backward difference feature sequence. Of course, those skilled in the art will understand that the difference feature sequence may also include only the forward difference feature sequence or the backward difference feature sequence, and this disclosure will not elaborate further on this.

[0156] As can be seen from the above, in this embodiment of the disclosure, by using the forward and backward difference features of the video stream sequence as a difference feature sequence, the forward and backward differences of the video stream content are simultaneously fused, fully integrating the differences and changes, which is more conducive to improving the accuracy of subsequent keyframe extraction.

[0157] After determining the image feature sequence and difference feature sequence of the video stream sequence as described above, the two can be fused. The following section combines... Figure 5 The process of fusion processing is explained.

[0158] like Figure 5 As shown, in some embodiments, the image processing method of this disclosure, in the process of fusing image feature sequences and differential feature sequences, includes:

[0159] S510. Determine the target weight of the differential feature sequence based on the differential change rate of the inter-frame images in the differential feature sequence.

[0160] S520. Based on the image feature sequence, target weights and difference feature sequences, perform fusion processing to obtain the first fused feature sequence.

[0161] Based on the foregoing Figure 4 As can be seen from the implementation method, the image feature sequence represents the original features of the video stream sequence, while the forward difference feature sequence and the backward difference feature sequence represent the difference features of the video stream sequence. The goal of the fusion processing in this implementation method is to fuse the difference features represented by the forward difference feature sequence and the backward difference feature sequence into the original features represented by the image feature sequence.

[0162] It's worth noting that different video streams exhibit varying degrees of content variation. Therefore, the weights corresponding to the difference feature sequences should also differ when performing difference feature fusion. Understandably, for video streams with minimal content variation, the weights of the difference feature sequences can be increased to enhance the expression of differences; conversely, for video streams with significant content variation, the weights of the difference feature sequences can be decreased to weaken the expression of differences.

[0163] Based on the above theoretical guidance, the weight values ​​corresponding to the differential feature sequences, i.e., the target weights, can be determined according to the differential change rate of the inter-frame images in the differential feature sequence. The differential change rate represents the temporal variability of consecutive frame images. The target weights are negatively correlated with the differential change rate; that is, the larger the differential change rate, the smaller the target weights.

[0164] Using the aforementioned example, the forward difference feature sequence is denoted as d1, and the backward difference feature sequence is denoted as d2. In some implementations, an LSTM network can be used to capture the temporal differential rates of change of the forward and backward difference feature sequences, denoted as vd1 and vd2, respectively. The temporal difference network can determine the corresponding weight values, i.e., the target weights, for the forward and backward difference feature sequences based on the magnitudes of vd1 and vd2, respectively.

[0165] After determining the target weights corresponding to the forward and backward difference feature sequences, the forward and backward difference feature sequences can be fused into the original image feature sequence based on these target weights to obtain the first fused feature sequence that incorporates the differential features.

[0166] As can be seen from the above, in this embodiment of the present disclosure, by fusing inter-frame difference features, the network can fully learn the temporal differences of the video stream sequence, especially for video streams with small content changes, thereby reducing redundant frames in the extracted keyframes and improving the accuracy of keyframe extraction.

[0167] After obtaining the first fused feature sequence, it can be processed based on the self-attention network 300 to fuse the correlation features between the fused frame images, thus obtaining the second fused feature sequence. The following section combines... Figure 6 Please provide an explanation.

[0168] like Figure 6 As shown, in some embodiments, the image processing method of this disclosure, which processes the first fused feature sequence according to a correlation weight, includes the following steps:

[0169] S610. Based on the self-attention mechanism, determine the correlation weights of inter-frame images.

[0170] S620. The first fusion feature sequence is processed according to the correlation weight to obtain the second fusion feature sequence.

[0171] The self-attention network 300 can extract correlation features between adjacent frames, thereby enabling the output second fusion feature sequence to further fuse inter-frame correlations.

[0172] In some embodiments of this disclosure, the self-attention network 300 can calculate the correlation matrix between adjacent frame images based on the input first fused feature sequence using a self-attention mechanism, and obtain correlation weights after processing such as softmax normalization. The correlation weights represent the correlation between the images in the frames. For example, the higher the correlation between two adjacent frame images, the larger their corresponding correlation weights; conversely, the lower the correlation between two adjacent frame images, the smaller their corresponding correlation weights.

[0173] After obtaining the correlation weight, the correlation weight can be multiplied by the first fusion feature sequence, thereby further fusing the correlation features in the features of each frame image of the first fusion feature sequence to obtain the second fusion feature sequence.

[0174] It is understood that the first fused feature sequence obtained by the aforementioned difference feature sequence fusion aggregates the difference features of inter-frame images. In this embodiment, the second fused feature sequence is further aggregated by correlation feature fusion, so that the second fused feature sequence simultaneously has the difference and correlation features of inter-frame images.

[0175] After obtaining the second fused feature sequence, it can be input into the pre-trained regression network 400 to obtain the importance score of each frame of the image output by the regression network 400. The following section combines... Figure 7 Please provide an explanation.

[0176] like Figure 7 As shown, in some embodiments, the image processing method of this disclosure, in the process of determining a keyframe image based on a second fused feature sequence, includes:

[0177] S710. Based on the second fusion feature sequence, the importance score of each frame image is predicted.

[0178] In some implementations, the regression network 400 can perform regression prediction based on the second fused feature sequence output by the self-attention network 300, and the output prediction result is the importance score of each frame in the video stream sequence.

[0179] It can be understood that the importance score represents the probability that each frame is a keyframe image. The higher the importance score, the higher the probability that the frame is a keyframe image, and vice versa.

[0180] In some implementations, the regression network 400 can collect network structures based on logistic regression. The specific structure and principles of the regression network can undoubtedly be understood and fully implemented by those skilled in the art by referring to relevant technologies, and this disclosure will not elaborate further.

[0181] S720. Perform image detection on the video stream sequence to determine at least one scene switching node in the video stream sequence.

[0182] S730. The video stream sequence is segmented based on at least one scene switching node to obtain at least two video sequence segments.

[0183] It is understandable that a video stream sequence often contains multiple scenes. For example, a street surveillance video might contain segments before the target vehicle appears, segments when the target vehicle appears, and segments after the target vehicle disappears. Each segment corresponds to a different scene stage. In other words, the corresponding scene of the video stream changes from the video sequence segment before the target vehicle appears to the video sequence segment when the target vehicle appears, and then changes again to the video sequence segment after the target vehicle disappears.

[0184] In this embodiment of the disclosure, image detection technology can be used to perform image detection on the video stream sequence to determine the scene change points of the video stream sequence. Scene change points represent the nodes in the video stream sequence where scene changes occur. For example, in the example above, the frame image at the moment the target vehicle appears and the frame image at the moment the target vehicle disappears are the scene change points.

[0185] After determining the scene transition nodes, each scene transition node can be used as a critical frame for segmenting the video stream sequence, thus dividing the video stream sequence into multiple video sequence segments. For example, in the example above, using the frame image at the moment the target vehicle appears and the frame image at the moment the target vehicle disappears as scene transition nodes, the video stream sequence can be segmented to obtain three video sequence segments.

[0186] S740. Determine the keyframe images based on the length of each video sequence segment and the importance score of each frame image in the video sequence segment.

[0187] In this embodiment of the disclosure, considering that if keyframe images are determined solely based on importance scores, video sequence segments with many redundant frame images may be identified as keyframe images, reducing the accuracy of keyframe extraction, the length and importance scores of each segmented video sequence segment are comprehensively considered to determine the keyframe images.

[0188] Specifically, based on prior knowledge or specific application scenario requirements, a threshold for the number of keyframe images can be pre-set. When the number of image frames contained in the segmented video sequence exceeds this threshold, it indicates that the length of the video sequence segment is too long and therefore it is not a keyframe image.

[0189] If the number of image frames contained in the segmented video sequence does not exceed the specified threshold, then the importance score of each frame in the video sequence segment is further obtained. When the importance scores of the frame images in the video sequence segment meet a preset condition, all frame images in the video sequence segment are determined to be keyframe images. For example, in one example, a score threshold can be preset. When the importance scores of all frame images in the video sequence segment exceed the score threshold, all frame images in the video sequence segment are determined to be keyframe images.

[0190] As can be seen from the above, in this embodiment of the disclosure, the key frame image is determined by combining the length of the video sequence segment and the importance score of each frame image, thereby further improving the accuracy of key frame extraction.

[0191] In some embodiments, the image processing method of this disclosure further includes the above-described... Figure 2 The image detection network shown below undergoes a training process, which will be discussed in conjunction with the following steps. Figure 8 Please provide an explanation.

[0192] like Figure 8 As shown, in some embodiments, the image processing method of this disclosure, in the process of training an image detection network, includes:

[0193] S810. Input the video stream sample sequence into the feature extraction network to be trained to obtain the image feature sequence output by the feature extraction network.

[0194] S820. Input the image feature sequence into the temporal difference network to be trained to obtain the first fused feature sequence output by the temporal difference network.

[0195] S830. Input the first fused feature sequence into the self-attention network to be trained to obtain the second fused feature sequence output by the self-attention network.

[0196] S840. Input the second fused feature sequence into the regression network to be trained to obtain the prediction result output by the regression network.

[0197] S850. Based on the prediction results and the differences between the score labels, adjust the parameters of at least one of the feature extraction network, temporal difference network, self-attention network and regression network to be trained until the convergence condition is met.

[0198] In some implementations, a sample dataset required for training can first be obtained. The sample dataset may include multiple sample data. For ease of explanation, the following description uses one sample data as an example.

[0199] The sample data may include video sample sequences, which consist of multiple temporally consecutive frames. Each frame in the video sample sequence has a score label and a keyframe label. The score label represents the standard value of the image's importance score, and the keyframe label represents the standard value indicating whether the image is a keyframe.

[0200] It is understandable that score labels and keyframe labels represent the ground truth of an image, and these labels can be manually annotated. In one example, keyframe labels can be represented by 0 and 1, where 0 represents a non-keyframe and 1 represents a keyframe.

[0201] For any sample data, the video stream sample sequence can be input into the feature extraction network 100 to be trained, so that the feature extraction network 100 can extract features from each frame of the image and obtain and output the image feature sequence corresponding to the video stream sample sequence.

[0202] The feature extraction network 100 outputs the image feature sequence to the temporal difference network 200 to be trained. Based on the aforementioned process, the temporal difference network 200 determines the corresponding difference feature sequence, and then fuses the difference feature sequence with the image feature sequence to obtain and output the first fused feature sequence.

[0203] The temporal difference network 200 outputs the first fused feature sequence to the self-attention network 300 to be trained. The self-attention network 300 determines the correlation weights based on the self-attention mechanism and processes the first fused feature sequence based on the correlation weights to obtain and output the second fused feature sequence.

[0204] The second fused feature sequence output from the attention network 300 is fed into the regression network 400 to be trained. The regression network outputs a prediction result based on the second fused feature sequence. The prediction result includes the importance score of each frame in the video stream sample sequence.

[0205] It can be understood that the importance score of each frame of image output by the regression network 400 is the predicted value, while the score label of each frame of image included in the sample data is the true value. Therefore, after obtaining the importance score of each frame of image output by the regression network 400, the difference between the importance score and the score label of each frame of image can be calculated based on a pre-constructed loss function. This difference is then backpropagated to each network to be trained, adjusting and optimizing at least one network parameter. For example, in one example, the MES regression loss function can be used to calculate the difference between the importance score and the score label of each frame of image.

[0206] The above describes the training process of an image detection network using a single sample dataset. During network training, a large amount of sample data can be used to train each of the aforementioned networks until the convergence condition is met, at which point network training is complete. For example, in one scenario, the image detection network can be trained 100 times using a sample dataset, and the network with the best performance can be used as the final trained network.

[0207] It's understandable that a relatively long video stream sequence often includes more than one keyframe segment. Taking street scene surveillance video as an example, the task of keyframe extraction is to extract the frames in which objects pass as keyframes. In this scenario, the video stream sequence may include multiple video segments such as "segments in which no objects pass," "segments in which a human body passes," "segments in which motor vehicles pass," and "segments in which non-motorized vehicles pass," where each segment in which an object (including human body, motor vehicle, and non-motorized vehicle) passes is a keyframe segment.

[0208] In this scenario, the inventors discovered through research that keyframe extraction methods in related technologies, because image detection networks only focus on the correlation between video frames, lose the differences in image content between frames. This results in the extracted keyframe segments having extremely similar video content and lacking diversity. For example, in the scenario mentioned above, as the image detection network in related technologies trains, the video keyframe segments extracted by the image detection network may only tend towards video sequences of people passing by, while neglecting to extract keyframes of segments of motor vehicles or non-motor vehicles passing by. This leads to the final extracted keyframe segments having monotonous content, insufficient diversity, and difficulty in practical application.

[0209] Based on this, in some embodiments of this disclosure, during the training of the image detection network, inter-frame constraints based on contrastive learning are used to enforce constraints on the differences between keyframe segments, thereby improving the diversity of keyframe segment extraction. The following describes how... Figure 9 Please provide an explanation.

[0210] like Figure 9 As shown, in some embodiments, the image processing method of this disclosure, in the training process of the image detection network, further includes:

[0211] S910, Obtain sample data.

[0212] S920. Based on the keyframe labels, determine a preset number of non-keyframe images adjacent to each keyframe segment as target frame segments.

[0213] In this embodiment of the disclosure, the sample dataset required for training can first be obtained. The sample dataset may include multiple sample data. For ease of explanation, the following description uses one sample data as an example.

[0214] The sample data may include video sample sequences, which consist of multiple temporally consecutive frames. Each frame in the video sample sequence has a score label and a keyframe label. For the score label and keyframe label, those skilled in the art refer to... Figure 8 The implementation method is self-explanatory, and this disclosure will not elaborate further.

[0215] In this embodiment, the video sample sequence includes at least two keyframe segments. For example, in one example, the video sample sequence is a street surveillance video, which includes the following 5 video segments in time sequence:

[0216] The first segment is a video clip with no objects passing by, ranging from frame 1 to frame 50.

[0217] The second segment is a video clip from the appearance to the disappearance of the human body, ranging from frame 51 to frame 100.

[0218] The third segment is a video clip with no objects passing by, ranging from frame 101 to frame 150.

[0219] The fourth segment is a video clip from the appearance to the disappearance of the vehicle, ranging from frame 151 to frame 200.

[0220] The fifth segment is a video clip with no objects passing by, ranging from frame 201 to frame 250.

[0221] Of the five video clips mentioned above, the second and fourth clips are keyframe clips, where all frames are keyframes and the corresponding keyframe label is 1. The first, third, and fifth clips are non-keyframe clips, where all frames are non-keyframes and the corresponding keyframe label is 0.

[0222] In this embodiment of the disclosure, the sample data first needs to be preprocessed to construct a data pair of <keyframe segment, target frame segment> corresponding to each keyframe segment. Using the example above as an example, for ease of understanding, the second segment is defined as "keyframe segment 1" and the fourth segment is defined as "keyframe segment 2".

[0223] For keyframe segment 1, a predetermined number of non-keyframe images preceding and / or following keyframe segment 1 in time sequence can be identified as target frame segments. These non-keyframe images can be determined based on the keyframe labels of the images. For example, the non-keyframe images preceding and preceding keyframe segment 1 (10 frames in total) can be identified as target frame segments. That is, the target frame segments corresponding to keyframe segment 1 include frames 41-50 and frames 101-110, and this target frame segment is defined as target frame segment 1.

[0224] Similarly, for keyframe segment 2, a predetermined number of non-keyframe images preceding and / or following keyframe segment 2 in time sequence can be identified as target frame segments. These non-keyframe images can be determined based on the keyframe labels of the images. For example, the non-keyframe images preceding and preceding keyframe segment 2 in the order of 10 frames can be identified as target frame segments. That is, the target frame segments corresponding to keyframe segment 2 include frames 141-150 and frames 201-210, and this target frame segment is defined as target frame segment 2.

[0225] Through the above data preprocessing, the data pairs corresponding to each keyframe segment in the sample data are obtained, represented as: <keyframe segment 1, target frame segment 1> and <keyframe segment 2, target frame segment 2>.

[0226] S930. For any keyframe segment, based on the image features of the keyframe segment and the image features of each target frame segment, determine the first similarity feature between the keyframe segment and the corresponding target frame segment, and the second similarity feature between the keyframe segment and each non-corresponding target frame segment.

[0227] S940. By comparing the ratio of the first similarity feature and the second similarity feature corresponding to each keyframe segment, the contrast loss between frames is determined.

[0228] S950. Based on the contrastive loss, adjust the parameters of at least one of the feature extraction network, temporal difference network, self-attention network, and regression network to be trained until the convergence condition is met.

[0229] based on Figure 2 As shown in the image detection network structure, for a video stream sample sequence, a second fused feature sequence that integrates inter-frame correlation and differences can be obtained based on the feature extraction network 100, the temporal difference network 200, and the self-attention network 300. Therefore, for video stream sample sequences such as keyframe segments and target frame segments, the image features of each frame image included can be obtained, and it is only necessary to extract the corresponding frame image from the second fused feature sequence.

[0230] In the example above, taking keyframe fragment 1 as an example, target frame fragment 1 is the target frame fragment corresponding to keyframe fragment 1, while target frame fragment 2 is the target frame fragment that does not correspond to keyframe fragment 1.

[0231] In this embodiment of the disclosure, the first similarity feature represents the correlation feature between a keyframe segment and its corresponding target frame segment. For example, the correlation feature between keyframe segment 1 and target frame segment 1 is the first similarity feature. The second similarity feature represents the correlation feature between a keyframe segment and its non-corresponding target frame segment. For example, the correlation feature between keyframe segment 1 and target frame segment 2 is the second similarity feature.

[0232] For any given keyframe segment, based on the image features of the keyframe segment and the image features of all target frame segments, the similarity features between each keyframe segment and each target frame segment are calculated. For example, in the example above, the first similarity feature between keyframe segment 1 and target frame segment 1, and the second similarity feature between keyframe segment 1 and target frame segment 2 can be calculated respectively.

[0233] In this embodiment of the disclosure, the inter-frame contrast loss L is determined by comparing the ratio of the first similarity feature to the second similarity feature corresponding to each keyframe segment. NCE , represented as:

[0234]

[0235] In equation (1), L NCE E represents the comparative loss. x Let f(·) represent the expectation, and v represent the similarity function. i Represents a keyframe segment, v j Indicates with v i The corresponding target frame segment, v k Indicates with v i Non-corresponding target frame fragment.

[0236] See equation (1) above, f(v) i ,v j f(v) represents the first similarity feature between a keyframe segment and its corresponding target frame segment. i ,v k ) represents the second similarity feature between a keyframe segment and its non-corresponding target frame segment.

[0237] Based on the optimization of the objective function, it can be seen that the objective task of network training should be to minimize the contrastive loss L. NCE To be as small as possible, so that for equation (1) above, the first similarity feature f(v) should be minimized. i ,v j ) and each second similarity feature The ratio of f(v) should be maximized; therefore, the training objective is to make f(v) such that f(v) = ... i ,v j Make f(v) as large as possible, so that f(v) i ,v k(As small as possible)

[0238] It is understandable that f(v) i ,v j f(v) represents the relevance between a keyframe segment and its corresponding target frame segment. The training objective is to maximize this relevance, meaning to ensure a higher relevance between the keyframe segment and its neighboring target frame segments, thus making the keyframe segment more representative. i ,v k The correlation between a keyframe segment and its non-corresponding target frame segment is represented by the number of keyframe segments. The training objective is to minimize this correlation, which means that the correlation between a keyframe segment and its non-adjacent target frame segments is lower, resulting in higher dissimilarity and greater differences between different keyframe segments, thus making multiple keyframe segments more diverse.

[0239] In this embodiment of the disclosure, the contrastive loss based on contrastive learning forces key frame segments to have a high correlation with adjacent frames, while key frame segments have a high difference with non-adjacent frame images. This makes the key frame images extracted by the network both representative and differentiated, greatly improving the key frame extraction effect.

[0240] For any sample data, the contrast loss L is calculated through the above process. NCE Then, the comparison loss L can be determined. NCE Backpropagation is performed on each network to be trained, and at least one network parameter is adjusted and optimized.

[0241] The above describes the training process of an image detection network using a single sample dataset. During network training, a large amount of sample data can be used to train each of the aforementioned networks until the convergence condition is met, at which point network training is complete. For example, in one scenario, the image detection network can be trained 100 times using a sample dataset, and the network with the best performance can be used as the final trained network.

[0242] It is understandable that, regarding the above Figure 9 The training process of the implementation method can be with Figure 8 The implementation methods are carried out simultaneously, thereby shortening the network training time and improving network training efficiency. Furthermore, for any aspects of the training process not detailed above, those skilled in the art can undoubtedly understand and fully implement them by referring to relevant technologies, and this disclosure will not elaborate further.

[0243] As described above, in this embodiment, based on the correlation between images in the video stream frames and combined with the differences between images in the frames, the extracted keyframe images simultaneously consider both correlation and differences. Especially for video stream sequences with minimal content variation, this effectively reduces redundant frames in the extracted video keyframes, improving the accuracy of keyframe extraction. Furthermore, by using both the forward and backward difference features of the video stream sequence as a difference feature sequence, the forward and backward differences of the video stream content are simultaneously integrated, fully incorporating the differences and improving the accuracy of keyframe extraction. Additionally, the contrastive loss based on contrastive learning enforces a high correlation between keyframe segments and adjacent frames, while maintaining a high difference between keyframe segments and non-adjacent frame images. This ensures that the keyframe images extracted by the network are both representative and differentiated, significantly improving the keyframe extraction effect.

[0244] Secondly, embodiments of this disclosure provide an image processing apparatus that can be applied to an electronic device and whose processing is performed by the processor of the electronic device. The electronic device described in embodiments of this disclosure can be any suitable type of device, such as user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, vehicle-mounted device, wearable device, etc., and this disclosure does not impose any limitations thereon.

[0245] like Figure 10 As shown, in some embodiments, the image processing apparatus of this disclosure includes:

[0246] Feature extraction module 10 is configured to extract features from video stream sequences to obtain image feature sequences of the video stream sequences and difference feature sequences for inter-frame images;

[0247] The differential fusion module 20 is configured to perform fusion processing based on the image feature sequence and the difference feature sequence to obtain the first fused feature sequence;

[0248] The self-attention module 30 is configured to process the first fused feature sequence according to the correlation weights of the inter-frame images to obtain the second fused feature sequence;

[0249] The keyframe determination module 40 is configured to determine at least one keyframe image from the video stream sequence based on a second fused feature sequence.

[0250] As can be seen from the above, in this embodiment of the present disclosure, based on the correlation between images between video stream frames and combined with the differences between images between frames, the extracted keyframe images take into account both the correlation and differences between frames. Especially for video stream sequences with small content changes, it can effectively reduce redundant frames in the extracted video keyframes and improve the accuracy of keyframe extraction.

[0251] In some implementations, the feature extraction module 10 is configured to:

[0252] Feature extraction is performed on the video stream sequence to obtain the image features of each frame in the video stream sequence;

[0253] Based on the image features of each frame, the image feature sequence of the video stream sequence is determined.

[0254] In some implementations, the feature extraction module 10 is configured to:

[0255] For any frame of an image in a video stream sequence, forward difference features are determined based on the image features of the image and the image features of the previous frame.

[0256] Based on the image features of the image and the image features of the next frame, the backward difference features are determined;

[0257] Based on the forward and backward difference features of all frames in the video stream sequence, the difference feature sequence of the video stream sequence is determined.

[0258] As can be seen from the above, in this embodiment of the disclosure, by using the forward and backward difference features of the video stream sequence as a difference feature sequence, the forward and backward differences of the video stream content are simultaneously fused, fully integrating the differences and changes, which is more conducive to improving the accuracy of subsequent keyframe extraction.

[0259] In some implementations, the differential fusion module 20 is configured as follows:

[0260] The target weights of the differential feature sequence are determined based on the rate of difference change between frames in the differential feature sequence; the rate of difference change is determined based on the differential features of each frame.

[0261] The first fused feature sequence is obtained by fusing the image feature sequence, target weights, and difference feature sequences.

[0262] As can be seen from the above, in this embodiment of the present disclosure, by fusing inter-frame difference features, the network can fully learn the temporal differences of the video stream sequence, especially for video streams with small content changes, thereby reducing redundant frames in the extracted keyframes and improving the accuracy of keyframe extraction.

[0263] In some implementations, the self-attention module 30 is configured as follows:

[0264] Based on the self-attention mechanism, the correlation weights of inter-frame images are determined;

[0265] The first fused feature sequence is processed according to the correlation weights to obtain the second fused feature sequence.

[0266] In some implementations, the keyframe determination module 40 is configured to:

[0267] Based on the second fusion feature sequence, the importance score of each frame of the image is predicted;

[0268] Perform image detection on the video stream sequence to determine at least one scene switching node in the video stream sequence;

[0269] The video stream sequence is segmented based on at least one scene switching node to obtain at least two video sequence segments;

[0270] Keyframe images are determined based on the length of each video sequence segment and the importance score of each frame in the video sequence segment.

[0271] As can be seen from the above, in this embodiment of the disclosure, the key frame image is determined by combining the length of the video sequence segment and the importance score of each frame image, thereby further improving the accuracy of key frame extraction.

[0272] In some implementations, the feature extraction module 10 is configured to: input a video stream sequence into a pre-trained feature extraction network to obtain an image feature sequence output by the feature extraction network; and input the image feature sequence into a pre-trained temporal difference network to obtain a difference feature sequence output by the temporal difference network.

[0273] The differential fusion module 20 is configured such that: the temporal difference network performs fusion processing based on the image feature sequence and the difference feature sequence, and outputs the first fused feature sequence;

[0274] The self-attention module 30 is configured to: input the first fused feature sequence into the pre-trained self-attention network, so that the self-attention network processes the first fused feature sequence according to the determined relevance weights and outputs the second fused feature sequence;

[0275] The keyframe determination module 40 is configured to input the second fused feature sequence into a pre-trained regression network to obtain the prediction result output by the regression network, which represents the prediction result of the keyframe image.

[0276] In some embodiments, the apparatus further includes a training module configured to:

[0277] Acquire sample data, which includes video stream sample sequences, score labels for each frame in the video stream sample sequences, and keyframe labels for each frame; the video stream sample sequences include at least two keyframe segments;

[0278] Based on the keyframe labels, a preset number of non-keyframe images adjacent to each keyframe segment are identified as target frame segments.

[0279] For any keyframe segment, based on the image features of the keyframe segment and the image features of each target frame segment, determine the first similarity feature between the keyframe segment and the corresponding target frame segment, and the second similarity feature between the keyframe segment and each non-corresponding target frame segment;

[0280] The contrast loss between frames is determined by comparing the ratio of the first similarity feature to the second similarity feature corresponding to each keyframe segment.

[0281] Based on the contrastive loss, at least one of the feature extraction network, temporal difference network, self-attention network, and regression network to be trained is adjusted in terms of parameters until the convergence condition is met.

[0282] In some implementations, the training module is configured as follows:

[0283] Input the video stream sample sequence into the feature extraction network to be trained to obtain the image feature sequence output by the feature extraction network;

[0284] The image feature sequence is input into the temporal difference network to be trained, and the first fused feature sequence output by the temporal difference network is obtained.

[0285] The first fused feature sequence is input into the self-attention network to be trained, and the second fused feature sequence output by the self-attention network is obtained.

[0286] The second fused feature sequence is input into the regression network to be trained, and the prediction result output by the regression network is obtained; the prediction result represents the importance score of each frame image in the video stream sample sequence;

[0287] Based on the prediction results and the differences between the score labels, at least one of the feature extraction network, temporal difference network, self-attention network, and regression network to be trained is adjusted in terms of parameters until the convergence condition is met.

[0288] As described above, in this embodiment, based on the correlation between images in the video stream frames and combined with the differences between images in the frames, the extracted keyframe images simultaneously consider both correlation and differences. Especially for video stream sequences with minimal content variation, this effectively reduces redundant frames in the extracted video keyframes, improving the accuracy of keyframe extraction. Furthermore, by using both the forward and backward difference features of the video stream sequence as a difference feature sequence, the forward and backward differences of the video stream content are simultaneously integrated, fully incorporating the differences and improving the accuracy of keyframe extraction. Additionally, the contrastive loss based on contrastive learning enforces a high correlation between keyframe segments and adjacent frames, while maintaining a high difference between keyframe segments and non-adjacent frame images. This ensures that the keyframe images extracted by the network are both representative and differentiated, significantly improving the keyframe extraction effect.

[0289] Thirdly, embodiments of this disclosure provide an electronic device, including:

[0290] Processor; and

[0291] The memory stores computer instructions that can be read by a processor, which, when the computer instructions are read, executes the method according to any embodiment of the first aspect.

[0292] Fourthly, embodiments of this disclosure provide a storage medium for storing computer-readable instructions for causing a computer to perform the method according to any embodiment of the first aspect.

[0293] Figure 11 A schematic diagram of the structure of an electronic device 600 suitable for implementing the method of this disclosure is shown. Figure 11 The electronic device shown can perform the corresponding functions of the processor and storage medium described above.

[0294] like Figure 11 As shown, the electronic device 600 includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a memory 602 or a program loaded into the memory 602 from a storage section 608. The memory 602 also stores various programs and data required for the operation of the electronic device 600. The processor 601 and the memory 602 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0295] The following components are connected to I / O interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to I / O interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 610 as needed so that computer programs read from it can be installed into storage section 608 as needed.

[0296] In particular, according to embodiments of this disclosure, the above-described method process can be implemented as a computer software program. For example, embodiments of this disclosure include a computer program product comprising a computer program tangibly embodied on a machine-readable medium, the computer program containing program code for performing the above-described methods. In such embodiments, the computer program can be downloaded and installed from a network via communication section 609, and / or installed from removable medium 611.

[0297] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0298] Obviously, the above embodiments are merely examples for clear illustration and are not intended to limit the embodiments. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all embodiments here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this disclosure.

Claims

1. An image processing method, characterized by, include: Feature extraction is performed on the video stream sequence to obtain the image feature sequence of the video stream sequence and the difference feature sequence for inter-frame images; A first fused feature sequence is obtained by fusing the image feature sequence and the difference feature sequence. The first fused feature sequence is processed based on the correlation weights of the inter-frame images to obtain the second fused feature sequence; Based on the second fused feature sequence, at least one keyframe image is determined from the video stream sequence; The process of fusing the image feature sequence and the difference feature sequence to obtain the first fused feature sequence includes: The target weight of the differential feature sequence is determined based on the differential change rate of the inter-frame images in the differential feature sequence. The differential change rate is determined based on the differential features of each frame image, and the target weight is negatively correlated with the differential change rate. The first fused feature sequence is obtained by fusing the image feature sequence, the target weight, and the difference feature sequence.

2. The method of claim 1, wherein, Feature extraction is performed on the video stream sequence to obtain the image feature sequence of the video stream sequence, including: Feature extraction is performed on the video stream sequence to obtain the image features of each frame in the video stream sequence; Based on the image features of each frame, the image feature sequence of the video stream sequence is determined.

3. The method according to claim 1, characterized in that, Feature extraction is performed on the video stream sequence to obtain the difference feature sequence for inter-frame images, including: For any frame of the video stream sequence, forward difference features are determined based on the image features of the image and the image features of the previous frame. Based on the image features of the image and the image features of the next frame, the backward difference features are determined. Based on the forward difference features and the backward difference features of all frame images in the video stream sequence, the difference feature sequence of the video stream sequence is determined.

4. The method according to claim 1, characterized in that, The step of processing the first fused feature sequence according to the correlation weights of the inter-frame images to obtain the second fused feature sequence includes: The correlation weights of the inter-frame images are determined based on the self-attention mechanism. The first fused feature sequence is processed according to the correlation weights to obtain the second fused feature sequence.

5. The method according to claim 1, characterized in that, Determining at least one keyframe image from the video stream sequence based on the second fused feature sequence includes: Based on the second fused feature sequence, the importance score of each frame image is predicted; Image detection is performed on the video stream sequence to determine at least one scene switching node in the video stream sequence; The video stream sequence is segmented based on the at least one scene switching node to obtain at least two video sequence segments; The keyframe image is determined based on the length of each video sequence segment and the importance score of each frame in the video sequence segment.

6. The method according to any one of claims 1 to 5, characterized in that, The step of extracting features from the video stream sequence to obtain the image feature sequence of the video stream sequence and the difference feature sequence for inter-frame images includes: The video stream sequence is input into a pre-trained feature extraction network to obtain the image feature sequence output by the feature extraction network; The image feature sequence is input into a pre-trained temporal difference network to obtain the difference feature sequence output by the temporal difference network. The process of fusing the image feature sequence and the difference feature sequence to obtain the first fused feature sequence includes: The temporal difference network performs fusion processing based on the image feature sequence and the difference feature sequence, and outputs the first fused feature sequence; The step of processing the first fused feature sequence according to the correlation weights of the inter-frame images to obtain the second fused feature sequence includes: The first fused feature sequence is input into a pre-trained self-attention network, so that the self-attention network processes the first fused feature sequence according to the determined relevance weights and outputs the second fused feature sequence. Determining at least one keyframe image from the video stream sequence based on the second fused feature sequence includes: The second fused feature sequence is input into a pre-trained regression network to obtain the prediction result output by the regression network, which represents the prediction result of the keyframe image.

7. The method according to claim 6, characterized in that, It also includes a training process for at least one of the feature extraction network, the temporal difference network, the self-attention network, and the regression network, the training process including: Acquire sample data, which includes a video stream sample sequence, a score label for each frame in the video stream sample sequence, and a keyframe label for each frame; the video stream sample sequence includes at least two keyframe segments; Based on the keyframe labels, a preset number of non-keyframe images adjacent to each keyframe segment are determined as target frame segments. For any keyframe segment, based on the image features of the keyframe segment and the image features of each target frame segment, a first similarity feature between the keyframe segment and the corresponding target frame segment, and a second similarity feature between the keyframe segment and each non-corresponding target frame segment are determined. The contrast loss between frames is determined by comparing the ratio of the first similarity feature to the second similarity feature corresponding to each keyframe segment. Based on the contrastive loss, at least one of the feature extraction network, the temporal difference network, the self-attention network, and the regression network to be trained is adjusted in terms of parameters until the convergence condition is met.

8. The method according to claim 7, characterized in that, The training process also includes: The video stream sample sequence is input into the feature extraction network to be trained to obtain the image feature sequence output by the feature extraction network; The image feature sequence is input into the temporal difference network to be trained to obtain the first fused feature sequence output by the temporal difference network; The first fused feature sequence is input into the self-attention network to be trained to obtain the second fused feature sequence output by the self-attention network; The second fused feature sequence is input into the regression network to be trained to obtain the prediction result output by the regression network; the prediction result represents the importance score of each frame image in the video stream sample sequence; Based on the prediction results and the differences between the score labels, at least one of the feature extraction network, the temporal difference network, the self-attention network, and the regression network to be trained is adjusted in terms of parameters until the convergence condition is met.

9. An image processing apparatus, characterized in that, include: The feature extraction module is configured to extract features from the video stream sequence to obtain an image feature sequence of the video stream sequence and a difference feature sequence for inter-frame images. The differential fusion module is configured to perform fusion processing based on the image feature sequence and the differential feature sequence to obtain a first fused feature sequence; The self-attention module is configured to process the first fused feature sequence according to the correlation weights of the inter-frame images to obtain the second fused feature sequence; The keyframe determination module is configured to determine at least one keyframe image from the video stream sequence based on the second fused feature sequence; The differential fusion module is configured as follows: The target weight of the differential feature sequence is determined based on the differential change rate of the inter-frame images in the differential feature sequence. The differential change rate is determined based on the differential features of each frame image, and the target weight is negatively correlated with the differential change rate. The first fused feature sequence is obtained by fusing the image feature sequence, the target weight, and the difference feature sequence.

10. An electronic device, characterized in that, include: processor; as well as The memory stores computer instructions that can be read by the processor, and when the computer instructions are read, the processor executes the method according to any one of claims 1 to 8.

11. A storage medium, characterized in that, Used to store computer-readable instructions for causing a computer to perform the method according to any one of claims 1 to 8.

Images