Feature fusion method and device, video jitter elimination method and device

Abstract

The present disclosure relates to a feature fusion method and device, and a video jitter elimination method and device. The method includes: based on the features of the current image frame in the video to be processed and the features of at least one reference image frame, obtaining at least one first similarity information, wherein the at least one reference image frame is a predetermined number of image frames in front of the current image frame in the video to be processed, and the at least one first similarity information is the similarity information between the current image frame and at least one reference image frame; based on the features of at least one reference image frame and the at least one first similarity information, obtaining the features of at least one reference image frame after being aligned with the current image frame; based on the aligned features and the features of the current image frame, obtaining the fused features of the current image frame.

Description

Technical Field

[0001] The present disclosure relates to the field of video processing, and in particular to a feature fusion method and device, and a video jitter elimination method and device. Background Art

[0002] Video is a kind of information that we are exposed to more frequently in our daily life. With the development of deep learning, users hope to understand the content of the video through deep learning, such as how many people are in the video, the trajectory of people's movement, etc. However, in some complex scenes, two similar image frames in the video will produce very different results. For example, two image frames look similar to the naked eye, but the former image frame can detect human limbs, while the latter image frame suddenly cannot detect them. This phenomenon is called "jitter", which will seriously affect the user's perception and other quantitative indicators.

[0003] In order to solve the above problems, the academic community proposed "time series feature fusion methods" based on image algorithms for processing, such as using some RNN / non local methods. These time series feature fusion methods can effectively alleviate the jitter problem, but these methods are only useful on academic data sets. They will encounter many problems in real scenes, mainly because a lot of data in real scenes have never appeared in academic data sets, resulting in poor algorithm effects. These models need to be retrained using data from real scenes. However, this training process requires the annotation of a lot of real scene video data, especially for the time series feature fusion module that requires a large amount of video data for training, which is extremely difficult. Summary of the invention

[0004] The present disclosure provides a feature fusion method and device, and a video jitter elimination method and device, so as to at least solve the problem that the feature fusion method in the related art requires a large amount of video data for training, resulting in high costs such as manpower.

[0005] According to a first aspect of an embodiment of the present disclosure, a feature fusion method is provided, comprising: obtaining at least one first similarity information based on features of a current image frame in a video to be processed and features of at least one reference image frame, wherein the at least one reference image frame is a predetermined number of image frames in front of the current image frame in the video to be processed, and the at least one first similarity information is similarity information between the current image frame and the at least one reference image frame; obtaining features of at least one reference image frame after alignment with the current image frame based on features of the at least one reference image frame and the at least one first similarity information; and obtaining fused features of the current image frame based on the aligned features and features of the current image frame.

[0006] Optionally, based on the aligned features and the features of the current image frame, fused features of the current image frame are obtained, including: performing dot multiplication of the aligned features and the features of the current image frame with the features of the current image frame, respectively, to obtain at least two second similarity information; multiplying the aligned features and the features of the current image frame with corresponding second similarity information in at least two second similarity information, respectively, to obtain at least two products; and summing the at least two products to obtain fused features of the current image frame.

[0007] Optionally, the aligned features and the features of the current image frame are respectively multiplied with corresponding second similarity information in at least two pieces of second similarity information to obtain at least two products, including: normalizing at least two pieces of second similarity information to obtain at least two pieces of normalized second similarity information; and multiplying the aligned features and the current image frame with corresponding second similarity information in at least two pieces of normalized second similarity information to obtain at least two products.

[0008] Optionally, obtaining fused features of the current image frame based on the aligned features and the features of the current image frame includes: obtaining fused features of the current image frame based on an average value of the aligned features and the features of the current image frame.

[0009] Optionally, based on features of at least one reference image frame and at least one first similarity information, features of at least one reference image frame after being aligned with the current image frame are obtained, including: scaling and normalizing at least one similarity information; multiplying the features of at least one reference image frame with at least one normalized first similarity information to obtain features of at least one reference image frame after being aligned with the current image frame.

[0010] Optionally, the features of the current image frame and the features of at least one reference image frame are features output from any layer except the output layer in a video processing model used to perform image processing on the video to be processed, and the fused features are input to the next layer of any layer in the video processing model.

[0011] According to a second aspect of an embodiment of the present disclosure, a video jitter elimination method is provided, comprising: inputting a video to be processed into a video processing model, obtaining features of a current image frame in the video to be processed output by any layer except the output layer in the video processing model and features of at least one reference image frame, wherein the at least one reference image frame is a predetermined number of image frames in front of the current image frame in the video to be processed; fusing the features of the current image frame with the features of the at least one reference image frame by the feature fusion method as described above, to obtain fused features of the current image frame; and inputting the fused features into the next layer of any layer in the video processing model, to obtain a processed video.

[0012] According to a third aspect of an embodiment of the present disclosure, a feature fusion device is provided, comprising: a similarity acquisition unit, configured to acquire at least one first similarity information based on features of a current image frame in a video to be processed and features of at least one reference image frame, wherein the at least one reference image frame is a predetermined number of image frames in front of the current image frame in the video to be processed, and the at least one first similarity information is similarity information between the current image frame and the at least one reference image frame; an alignment unit, configured to obtain features of at least one reference image frame after alignment with the current image frame based on features of the at least one reference image frame and the at least one first similarity information; and a fused feature acquisition unit, configured to obtain fused features of the current image frame based on the aligned features and features of the current image frame.

[0013] Optionally, the fusion feature acquisition unit is further configured to perform dot multiplication of the aligned features and the features of the current image frame with the features of the current image frame, respectively, to obtain at least two second similarity information; multiply the aligned features and the features of the current image frame with corresponding second similarity information in the at least two second similarity information, respectively, to obtain at least two products; and sum the at least two products to obtain the fused features of the current image frame.

[0014] Optionally, the fusion feature acquisition unit is also configured to normalize at least two pieces of second similarity information to obtain at least two pieces of normalized second similarity information; and multiply the aligned features and the current image frame by the corresponding pieces of second similarity information in the at least two pieces of normalized second similarity information to obtain at least two products.

[0015] Optionally, the fused feature acquisition unit is further configured to obtain the fused features of the current image frame based on an average value of the aligned features and the features of the current image frame.

[0016] Optionally, the alignment unit is further configured to scale and normalize at least one similarity information; multiply features of at least one reference image frame by at least one normalized first similarity information to obtain features of at least one reference image frame aligned with the current image frame.

[0017] Optionally, the features of the current image frame and the features of at least one reference image frame are features output from any layer except the output layer in a video processing model used to perform image processing on the video to be processed, and the fused features are input to the next layer of any layer in the video processing model.

[0018] According to four aspects of the embodiments of the present disclosure, a video jitter elimination device is provided, including: a feature acquisition unit, configured to input a video to be processed into a video processing model, and obtain features of a current image frame in the video to be processed output by any layer except the output layer in the video processing model and features of at least one reference image frame, wherein the at least one reference image frame is a predetermined number of image frames in front of the current image frame in the video to be processed; a feature fusion unit, configured to fuse the features of the current image frame and the features of the at least one reference image frame by the feature fusion method as described above, and obtain fused features of the current image frame; and a processing unit, configured to input the fused features into a next layer of any layer in the video processing model, and obtain a processed video.

[0019] According to a fifth aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor executable instructions; wherein the processor is configured to execute instructions to implement a feature fusion method and / or a video jitter elimination method according to the present disclosure.

[0020] According to a sixth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to execute the feature fusion method and / or video jitter elimination method according to the present disclosure.

[0021] According to a seventh aspect of an embodiment of the present disclosure, a computer program product is provided, including computer instructions, which, when executed by a processor, implement the feature fusion method and / or video jitter elimination method according to the present disclosure.

[0022] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0023] According to the feature fusion method and device, and the video jitter elimination method and device disclosed in the present invention, a feature fusion method is provided, which obtains the first similarity information of the feature of the current image frame in the video to be processed and the feature of at least one reference image frame, wherein the at least one reference image frame is a predetermined number of image frames in front of the current image frame in the video to be processed, and then based on the feature of the at least one reference image frame and the at least one first similarity information, obtain the feature of at least one reference image frame after being aligned with the current image frame, and then based on the aligned feature and the feature of the current image frame, obtain the fused feature of the current image frame. It can be seen that the feature fusion method disclosed in the present invention does not need to label video data or perform training, and can be directly nested in the trained model to achieve feature fusion, reducing labor costs. In addition, the feature fusion method disclosed in the present invention obtains the fused feature of the current image frame based on the similarity information of the feature of the current image frame and the feature of at least one reference image frame. The fused feature well considers the feature of the image frame in front of the current graphic frame, which is equivalent to smoothing all the image frames of the video, and the jitter phenomenon in the related art will not occur. Therefore, after introducing the feature fusion method in the video processing process, the jitter problem in the video processing process can also be well alleviated. Therefore, the present disclosure solves the problem in the related art that the feature fusion method requires a large amount of video data for training, resulting in high costs such as manpower.

[0024] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0026] Figure 1 is a schematic diagram showing an implementation scenario of a feature fusion method according to an exemplary embodiment of the present disclosure;

[0027] Figure 2 is a flow chart of a feature fusion method according to an exemplary embodiment;

[0028] Figure 3 is a schematic diagram showing a feature alignment according to an exemplary embodiment;

[0029] Figure 4 is a schematic diagram of inserting a model using a fusion feature method according to an exemplary embodiment;

[0030] Figure 5 is a flow chart of a method for eliminating video jitter according to an exemplary embodiment;

[0031] Figure 6 is a schematic diagram showing a verification result according to an exemplary embodiment;

[0032] Figure 7 is a block diagram of a feature fusion device according to an exemplary embodiment;

[0033] Figure 8 is a block diagram of a video jitter elimination device according to an exemplary embodiment;

[0034] Fig. 9 is a block diagram of an electronic device 900 according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0035] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0036] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation methods described in the following examples do not represent all implementation methods consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the attached claims.

[0037] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.

[0038] The present invention provides a feature fusion method and a video jitter elimination method, which do not require the annotation of video data or training. Feature fusion can be achieved by directly embedding it in a trained model, and the jitter problem in video processing can also be well alleviated. The following is an example of a scene of face segmentation in a video to illustrate.

[0039] Figure 1is a schematic diagram showing an implementation scenario of a feature fusion method according to an exemplary embodiment of the present disclosure, such as Figure 1 The implementation scenario includes a server 100, a user terminal 110 and a user terminal 120, wherein the user terminals are not limited to two, and include but are not limited to mobile phones, personal computers and other devices. The user terminal can be equipped with a camera for acquiring facial images. The server can be a single server, or a server cluster composed of several servers, or a cloud computing platform or a virtualization center.

[0040] The server 100 receives the video to be segmented for human face sent by the user terminal 110 or 120. After receiving the video for human face segmentation, the server 100 inputs the video into the trained video segmentation model. The features output by any layer except the output layer in the video segmentation model can be used as the feature fusion method for applying the present disclosure. Assuming that the features output by the Nth layer of the video segmentation model are taken as an example, for each image frame output by the Nth layer: based on the features of the current image frame and the features of at least one reference image frame, at least one first similarity information is obtained, wherein at least one reference image frame is a predetermined number of image frames in front of the current image frame in the video to be segmented for human face, and at least one first similarity information is the similarity information between the current image frame and at least one reference image frame respectively; based on the features of at least one reference image frame and at least one first similarity information, the features of at least one reference image frame after being aligned with the current image frame are obtained; based on the aligned features and the features of the current image frame, the fused features of the current image frame are obtained. After the fused features are obtained, they are input into the N+1th layer, so that the fusion of the features output by the Nth layer is completed. The fused features are input into the next layer of any layer in the video segmentation model to obtain the segmented video. Since the present invention obtains the fused features of the current image frame based on the similarity information between the features of the current image frame and the features of at least one reference image frame, the fused features take into account the features of the image frame before the current graphic frame. Therefore, the segmented video can well avoid the jitter problem that occurs in video processing.

[0041] Hereinafter, a feature fusion method and apparatus, and a video jitter elimination method and apparatus according to exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

[0042] Figure 2 is a flow chart of a feature fusion method according to an exemplary embodiment. Figure 2 As shown, the feature fusion method includes the following steps:

[0043] In step S201, at least one first similarity information is obtained based on the features of the current image frame in the video to be processed and the features of at least one reference image frame, wherein the at least one reference image frame is a predetermined number of image frames in front of the current image frame in the video to be processed, and the at least one first similarity information is the similarity information between the current image frame and the at least one reference image frame. Generally, the features are in the form of a matrix, and the first similarity information can also be in the form of a matrix. For example, the way to obtain at least one first similarity information can be to multiply the features of the current image frame in the video to be processed with the features of at least one reference image frame, respectively, to obtain at least one first similarity matrix, and each first similarity matrix represents the similarity between the positions of the feature graphs of the current image frame and the corresponding reference image frame.

[0044] return Figure 2 In step S202, based on the features of at least one reference image frame and at least one first similarity information, the features of at least one reference image frame after being aligned with the current image frame are obtained. The aligned features can also be aligned using other alignment methods, such as optical flow alignment, which is not limited in the present disclosure.

[0045] According to an exemplary embodiment of the present disclosure, based on the features of at least one reference image frame and at least one first similarity information, the features of at least one reference image frame after being aligned with the current image frame are obtained, including: scaling and normalizing at least one similarity information; multiplying the features of at least one reference image frame with at least one first similarity information after normalization, to obtain the features of at least one reference image frame after being aligned with the current image frame. According to this embodiment, the subsequent fusion ratio of the reference image frame and the current image frame is controlled by scaling and normalization, ensuring that the fused features are more in line with the requirements, so that the fusion effect is better.

[0046] For example, Figure 3 is a schematic diagram showing a feature alignment according to an exemplary embodiment, such as Figure 3 As shown, the F matrix is ​​the feature of the current image frame, and the F1 matrix is ​​the feature of the image frame before the current image frame (corresponding to the above-mentioned reference image frame). The F matrix and the F1 matrix are multiplied to calculate the similarity of the two feature maps at each position, that is, to obtain the first similarity matrix, and then the obtained first similarity matrix is ​​scaled and normalized (such as softmax). Then, the F1 feature is matrix-multiplied with the first similarity matrix after softmax, and the calculated result is the aligned feature warp_F1. The same process is performed on other features F2...FN (other reference image frames) to obtain warp_F2...warp_FN. The scale of the above scale change can be set as needed.

[0047] return Figure 2 In step S203, based on the aligned features and the features of the current image frame, the fused features of the current image frame are obtained. It should be noted that there are many ways to obtain the fused features of the current image frame based on the aligned features and the features of the current image frame. It is sufficient to fuse the aligned features with the features of the current image frame. For example, the average value of the above features can be taken as the fused features. The similarity between the aligned features and the current image frame can also be obtained as the fusion weight, and the fusion is performed according to the weight. Of course, other methods are also possible, and the present disclosure does not limit this.

[0048] The following two methods are mainly introduced: 1) obtaining the similarity between the aligned features and the current image frame as the fusion weight, and fusing according to the weight; 2) taking the average value of the above features as the fused feature.

[0049] According to an exemplary embodiment of the present disclosure, based on the aligned features and the features of the current image frame, the fused features of the current image frame are obtained, including: performing dot multiplication of the aligned features and the features of the current image frame with the features of the current image frame, respectively, to obtain at least two second similarity information; performing multiplication of the aligned features and the features of the current image frame with the corresponding second similarity information in the at least two second similarity information, respectively, to obtain at least two products; summing the at least two products to obtain the fused features of the current image frame. According to this embodiment, obtaining the similarity between the aligned features and the current image frame as the proportion of fusion, and performing fusion according to the proportion, that is, obtaining the fused features by weighted summation, can bring about a better fusion effect.

[0050] For example, first calculate the similarity matrix of warp_F1 and F, that is, the second similarity information: SIM_F1 = dot(F, warp_F1), where SIM_F1 dimension (shape) = (H, W, 1), F and warp_F1 dimension (shape) = (H, W, D), and the specific calculation method is to perform dot multiplication on the two features (warp_F1 and F). Similarly, for the features after other image frames are aligned (warp_F2...warp_FN), the same calculation method is used to obtain SIM_F2...SIM_FN, where SIM_F = dot(F, F). It should be noted that H and W are the matrix dimensions, and D is the dimension of each value in the matrix, such as the value in the matrix can be 1, can be {1, 1}, etc. Then, the similarity matrix is ​​used as the weight of the aligned features and the features of the current image frame, and a weighted sum is performed to obtain the fused features. For example, the fused features may be: new_F=SIM_F*F+SIM_F1*warp_F1+…SIM_FN*warp_FN.

[0051] According to an exemplary embodiment of the present disclosure, the aligned features and the features of the current image frame are respectively multiplied with the corresponding second similarity information in at least two second similarity information to obtain at least two products, including: normalizing the at least two second similarity information to obtain at least two normalized second similarity information; multiplying the aligned features and the current image frame with the corresponding second similarity information in at least two normalized second similarity information to obtain at least two products. According to this embodiment, the proportion of the reference image frame and the current image frame in subsequent fusion is controlled by normalization processing to ensure that the fused features are more in line with the requirements, so that the fusion effect is better.

[0052] For example, before weighted summation, SIM_F, SIM_F1, SIM_F2…FIM_FN may be normalized, and the normalization method may use the average L1 norm / L2 norm / softmax method, which is not limited in the present disclosure. For example, taking the L1 norm method as an example, the second similarity information SIM_F1' after normalization is SIM_F1 / (SIM_F+SIM_F1+SIM_F2+…SIM_FN), and SIM_F2'…SIM_FN' may be obtained in the same way. Then, the normalized similarity is weighted summed, and the fused feature is: new_F=SIM_F'*F+SIM_F1'*warp_F1+…+SIM_FN'*warp_FN.

[0053] According to an exemplary embodiment of the present disclosure, based on the aligned features and the features of the current image frame, the fused features of the current image frame are obtained, including: based on the average value of the aligned features and the features of the current image frame, the fused features of the current image frame are obtained. According to this embodiment, the fused features are obtained by taking the average value, which can ensure that the fused features are obtained at the fastest speed. For example, L1 Norm / L2 Norm / softmax, etc. can be used.

[0054] According to an exemplary embodiment of the present disclosure, the features of the current image frame and the features of at least one reference image frame are features output from any layer other than the output layer in a video processing model for image processing of the video to be processed, and the fused features are input to the next layer of any layer in the video processing model. According to this embodiment, the feature fusion method of the present disclosure can be plug-and-play, that is, it can be directly nested on a trained model.

[0055] For example, Figure 4 is a schematic diagram of inserting a model using a fusion feature method according to an exemplary embodiment. Figure 4 As shown, the network framework takes U2NET as an example (the U2NET network model is on the left, which can be replaced by other network models), and the multi-frame fusion module on the right is the feature fusion method disclosed in the present invention, that is, the feature F of the current image frame can be taken from the output features of any layer in the middle of the model, and fused with the features F1, F2...FN corresponding to the previous few image frames of the current image frame, and the fused feature new_F replaces the original feature F and is input into the subsequent network of the model.

[0056] It should be noted that the above network architecture can be replaced by any other network, and the present disclosure does not limit this. The above feature fusion method can be used in one layer of the model, or in multiple layers at the same time, that is, each layer performs a feature fusion process, and the specific number of layers selected for feature fusion processing can be determined based on the balance between processing speed and accuracy.

[0057] The above is the overall framework of the present disclosure. The number of reference image frames selected to assist in feature fusion can be controlled according to the balance between the required speed and accuracy. If the stability requirement is very high, a large N (such as 20) can be selected. If the stability requirement is not high, then a normal size N (such as 5) can be selected. In addition, the above feature F can be the output of any layer of the network model, and multiple layers of F can also be obtained, that is, multiple layers are subjected to feature fusion processing. In addition, if the edge stability is desired to be very high, feature F can be the output feature of the layer close to the network output position. For example, the output feature of the second to last layer is selected as feature F, so that the edge stability will be very high. If the edge and the entire limb stability are desired to be very high, the second to last, fourth, and eighth layers can be selected at the same time for feature fusion processing. However, this will affect the speed of the network model, so the specific selection of which layer's output feature can be balanced according to accuracy and speed.

[0058] Figure 5 FIG. 1 is a flow chart of a method for eliminating video jitter according to an exemplary embodiment. Figure 5 , the video jitter elimination method includes the following steps:

[0059] In step S501, the video to be processed is input into a video processing model to obtain features of a current image frame in the video to be processed output by any layer except the output layer in the video processing model and features of at least one reference image frame, wherein the at least one reference image frame is a predetermined number of image frames in front of the current image frame in the video to be processed.

[0060] In step S502, the features of the current image frame and the features of at least one reference image frame are fused by the feature fusion method described above to obtain fused features of the current image frame.

[0061] For example, based on features of a current image frame in a video to be processed and features of at least one reference image frame, at least one first similarity information is obtained, wherein the at least one reference image frame is a predetermined number of image frames in front of the current image frame in the video to be processed, and the at least one first similarity information is similarity information between the current image frame and at least one reference image frame; based on features of at least one reference image frame and at least one first similarity information, features of at least one reference image frame aligned with the current image frame are obtained; based on the aligned features and features of the current image frame, fused features of the current image frame are obtained.

[0062] In step S503, the fused features are input into the next layer of any layer in the video processing model to obtain a processed video.

[0063] In order to verify the feasibility of the present invention, an image segmentation task is taken as an example for verification. Figure 6 is a schematic diagram showing a verification result according to an exemplary embodiment. Figure 6 As shown, the first column is the original image, the second column is the result without introducing the method of the present invention, and the third column is the result with introducing the method of the present invention. It can be seen that there is a false detection in the middle column, while there is no false detection after introducing the method of the present invention, because the method of the present invention performs time series smoothing, which can suppress these occasional false detections.

[0064] In summary, the feature fusion method disclosed in the present invention does not need to label a large amount of real-scene video data. It can achieve good results by using a model trained in a real scene to embed the feature fusion method disclosed in the present invention. On the basis of alleviating the problem of jitter, it does not introduce any parameters that need to be trained. It is plug-and-play and can be embedded in any existing trained network model. At the same time, the present invention can control the balance between speed and accuracy by adjusting the number of frames N used and the number of layers of features F for feature fusion processing. In contrast, after STM training, only a fixed number of frames and fixed features can be used. The method disclosed in the present invention gives users a lot of room for adjustment.

[0065] Figure 7 is a block diagram of a feature fusion device according to an exemplary embodiment. Figure 7 , the device comprises:

[0066] The similarity acquisition unit 70 is configured to acquire at least one first similarity information based on the features of the current image frame in the video to be processed and the features of at least one reference image frame, wherein the at least one reference image frame is a predetermined number of image frames in front of the current image frame in the video to be processed, and the at least one first similarity information is the similarity information between the current image frame and the at least one reference image frame; the alignment unit 72 is configured to obtain the features of at least one reference image frame after alignment with the current image frame based on the features of the at least one reference image frame and the at least one first similarity information; the fusion feature acquisition unit 74 is configured to obtain the fused features of the current image frame based on the aligned features and the features of the current image frame.

[0067] According to an exemplary embodiment of the present disclosure, the fused feature acquisition unit 74 is further configured to perform dot multiplication of the aligned features and the features of the current image frame with the features of the current image frame, respectively, to obtain at least two second similarity information; multiply the aligned features and the features of the current image frame with the corresponding second similarity information in the at least two second similarity information, respectively, to obtain at least two products; and sum the at least two products to obtain the fused features of the current image frame.

[0068] According to an exemplary embodiment of the present disclosure, the fusion feature acquisition unit 74 is further configured to normalize at least two pieces of second similarity information to obtain at least two pieces of normalized second similarity information; and multiply the aligned features and the current image frame by the corresponding pieces of second similarity information in the at least two pieces of normalized second similarity information to obtain at least two products.

[0069] According to an exemplary embodiment of the present disclosure, the fused feature acquisition unit 74 is further configured to obtain the fused features of the current image frame based on an average value of the aligned features and the features of the current image frame.

[0070] According to an exemplary embodiment of the present disclosure, the alignment unit 72 is further configured to scale and normalize at least one similarity information; multiply the features of at least one reference image frame by at least one normalized first similarity information, to obtain features of at least one reference image frame after alignment with the current image frame.

[0071] According to an exemplary embodiment of the present disclosure, features of the current image frame and features of at least one reference image frame are features output from any layer except the output layer in a video processing model used to perform image processing on the video to be processed, and the fused features are input to the next layer of any layer in the video processing model.

[0072] Figure 8 is a block diagram of a video jitter elimination device according to an exemplary embodiment. Figure 8 , the device comprises:

[0073] The feature acquisition unit 80 is configured to input the video to be processed into the video processing model, and obtain the features of the current image frame and the features of at least one reference image frame in the video to be processed output by any layer except the output layer in the video processing model, wherein the at least one reference image frame is a predetermined number of image frames in front of the current image frame in the video to be processed; the feature fusion unit 82 is configured to fuse the features of the current image frame and the features of at least one reference image frame by the feature fusion method as described above, and obtain the fused features of the current image frame; the processing unit 84 is configured to input the fused features into the next layer of any layer in the video processing model, and obtain the processed video.

[0074] According to an embodiment of the present disclosure, an electronic device may be provided. Fig. 9It is a block diagram of an electronic device 900 according to an embodiment of the present disclosure, and the electronic device includes at least one memory 901 and at least one processor 902, wherein a set of computer executable instructions is stored in the at least one memory, and when the computer executable instruction set is executed by the at least one processor, a feature fusion method and / or a video jitter elimination method according to an embodiment of the present disclosure is executed.

[0075] As an example, the electronic device 900 may be a PC, a tablet device, a personal digital assistant, a smart phone, or other device capable of executing the above-mentioned instruction set. Here, the electronic device 1000 is not necessarily a single electronic device, but may also be any device or circuit capable of executing the above-mentioned instructions (or instruction sets) individually or in combination. The electronic device 900 may also be part of an integrated control system or a system manager, or may be configured as a portable electronic device interconnected with a local or remote (e.g., via wireless transmission) interface.

[0076] In the electronic device 900, the processor 902 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not limitation, the processor 902 may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0077] The processor 902 may execute instructions or codes stored in the memory, wherein the memory 901 may also store data. Instructions and data may also be sent and received over a network via a network interface device, wherein the network interface device may employ any known transmission protocol.

[0078] The memory 901 may be integrated with the processor 902, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. In addition, the memory 901 may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory 901 and the processor 902 may be operationally coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor 902 can read files stored in the memory 901.

[0079] In addition, the electronic device 900 may further include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device may be connected to each other via a bus and / or a network.

[0080] According to an embodiment of the present disclosure, a computer-readable storage medium may also be provided, wherein when the instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to execute the feature fusion method and / or the video jitter elimination method of the embodiment of the present disclosure. Examples of computer-readable storage media here include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0081] According to an embodiment of the present disclosure, a computer program product is provided, including computer instructions, which, when executed by a processor, implement the feature fusion method and / or video jitter elimination method of the embodiment of the present disclosure.

[0082] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0083] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A feature fusion method, characterized in that: include: Based on the features of the current image frame and the features of at least one reference image frame in the video to be processed, obtaining at least one first similarity information, wherein the at least one reference image frame is a predetermined number of image frames in the video to be processed that are before the current image frame, and the at least one first similarity information is the similarity information between the current image frame and the at least one reference image frame; Based on the features of the at least one reference image frame and the at least one first similarity information, obtaining features of the at least one reference image frame after being aligned with the current image frame respectively; Based on the aligned features and the features of the current image frame, obtaining fused features of the current image frame; Among them, based on the features of the current image frame in the video to be processed and the features of at least one reference image frame, at least one first similarity information is obtained, including: multiplying the features of the current image frame in the video to be processed with the features of the at least one reference image frame respectively, to obtain at least one first similarity matrix, each first similarity matrix represents the similarity between the pairwise positions of the feature map of the current image frame and the corresponding reference image frame.

2. The feature fusion method according to claim 1, characterized in that: The obtaining, based on the aligned features and the features of the current image frame, fused features of the current image frame includes: Performing dot multiplication of the aligned features and the features of the current image frame with the features of the current image frame, respectively, to obtain at least two second similarity information; Multiplying the aligned features and the features of the current image frame with corresponding second similarity information of the at least two second similarity information respectively to obtain at least two products; The at least two products are summed to obtain fused features of the current image frame.

3. The feature fusion method according to claim 2, characterized in that: The step of multiplying the aligned features and the features of the current image frame with corresponding second similarity information of the at least two second similarity information to obtain at least two products includes: Normalizing the at least two pieces of second similarity information to obtain at least two pieces of normalized second similarity information; The aligned features and the current image frame are respectively multiplied with corresponding second similarity information in the at least two normalized second similarity information to obtain at least two products.

4. The feature fusion method according to claim 1, characterized in that: The obtaining, based on the aligned features and the features of the current image frame, fused features of the current image frame includes: Based on the average value of the aligned features and the features of the current image frame, the fused features of the current image frame are obtained.

5. The feature fusion method according to claim 1, characterized in that: The obtaining, based on the feature of the at least one reference image frame and the at least one first similarity information, features of the at least one reference image frame after being aligned with the current image frame, comprises: Performing a scale change and normalization processing on the at least one similarity information; The features of the at least one reference image frame are respectively multiplied by at least one normalized first similarity information to obtain features of the at least one reference image frame after being aligned with the current image frame.

6. The feature fusion method according to any one of claims 1 to 5, characterized in that: The features of the current image frame and the features of the at least one reference image frame are features output from any layer except the output layer in a video processing model used to perform image processing on the video to be processed, and the fused features are input to the next layer of any layer in the video processing model.

7. A video jitter elimination method, characterized in that: include: Inputting the video to be processed into the video processing model, obtaining features of a current image frame in the video to be processed and features of at least one reference image frame output by any layer except the output layer in the video processing model, wherein the at least one reference image frame is a predetermined number of image frames in the video to be processed that are before the current image frame; fusing the features of the current image frame and the features of the at least one reference image frame by the feature fusion method according to any one of claims 1 to 6, to obtain fused features of the current image frame; The fused features are input into the next layer of any layer in the video processing model to obtain a processed video.

8. A feature fusion device, characterized in that: include: a similarity acquisition unit, configured to acquire at least one first similarity information based on a feature of a current image frame and a feature of at least one reference image frame in the video to be processed, wherein the at least one reference image frame is a predetermined number of image frames in the video to be processed that are before the current image frame, and the at least one first similarity information is similarity information between the current image frame and the at least one reference image frame; an alignment unit, configured to obtain features of the at least one reference image frame after being aligned with the current image frame based on the features of the at least one reference image frame and the at least one first similarity information; a fused feature acquisition unit, configured to obtain fused features of the current image frame based on the aligned features and the features of the current image frame; Among them, the similarity acquisition unit is configured to multiply the features of the current image frame in the video to be processed with the features of the at least one reference image frame respectively, so as to obtain at least one first similarity matrix, each of which represents the similarity between the feature maps of the current image frame and the corresponding reference image frame at two positions.

9. The feature fusion device according to claim 8, characterized in that: The fused feature acquisition unit is further configured to perform dot multiplication on the aligned features and the features of the current image frame, respectively, with the features of the current image frame, to obtain at least two second similarity information; perform multiplication on the aligned features and the features of the current image frame, respectively, with corresponding second similarity information in the at least two second similarity information, to obtain at least two products; and sum the at least two products to obtain the fused features of the current image frame.

10. The feature fusion device according to claim 9, characterized in that: The fusion feature acquisition unit is also configured to normalize the at least two second similarity information to obtain at least two normalized second similarity information; multiply the aligned features and the current image frame with the corresponding second similarity information in the at least two normalized second similarity information to obtain at least two products.

11. The feature fusion device according to claim 8, characterized in that: The fused feature acquisition unit is further configured to obtain the fused features of the current image frame based on an average value of the aligned features and the features of the current image frame.

12. The feature fusion device according to claim 8, characterized in that: The alignment unit is further configured to scale and normalize the at least one similarity information; multiply the features of the at least one reference image frame by the at least one normalized first similarity information to obtain the features of the at least one reference image frame after alignment with the current image frame.

13. The feature fusion device according to any one of claims 8 to 12, characterized in that: The features of the current image frame and the features of the at least one reference image frame are features output from any layer except the output layer in a video processing model used to perform image processing on the video to be processed, and the fused features are input to the next layer of any layer in the video processing model.

14. A video jitter elimination device, characterized in that: include: a feature acquisition unit, configured to input the video to be processed into the video processing model, and obtain features of a current image frame in the video to be processed and features of at least one reference image frame output by any layer except the output layer in the video processing model, wherein the at least one reference image frame is a predetermined number of image frames in the video to be processed that are before the current image frame; a feature fusion unit, configured to fuse the features of the current image frame and the features of the at least one reference image frame by using the feature fusion method according to any one of claims 1 to 6, so as to obtain fused features of the current image frame; The processing unit is configured to input the fused features into the next layer of any layer in the video processing model to obtain a processed video.

15. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the feature fusion method according to any one of claims 1 to 6 and / or the video jitter elimination method according to claim 7.

16. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to perform the feature fusion method according to any one of claims 1 to 6 and / or the video jitter elimination method according to claim 7.

17. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the feature fusion method according to any one of claims 1 to 6 and / or the video jitter elimination method according to claim 7 are implemented.

Images