Video processing method and device, and electronic device

Abstract

The application provides a video processing method and device, and an electronic device. A specific embodiment of the method includes: obtaining a to-be-processed video captured from at least two perspectives; for any to-be-processed video frame, calculating monocular similarity between the to-be-processed video frame and corresponding features of a plurality of adjacent to-be-processed video frames, and supplementing feature information of the to-be-processed video frame by using the monocular similarity to obtain supplemented features; for any two to-be-processed video frames captured from different perspectives at the same time point, calculating binocular similarity between a pair of corresponding supplemented features of the two to-be-processed video frames based on prior knowledge, and fusing the pair of supplemented features by using the binocular similarity; and performing super-resolution processing on the to-be-processed video based on the fused feature pair to obtain a target video. The method can improve phenomena such as artifacts and occlusion of the to-be-processed video after super-resolution processing, and effectively improves the video quality.

Description

Technical Field

[0001] This application relates to the field of information processing, and more specifically, to a video processing method, apparatus, and electronic device. Background Technology

[0002] Binocular stereo vision is an important form of machine vision. It is based on the principle of parallax and uses imaging equipment to acquire two images of the object being measured from different positions. By calculating the positional deviation between corresponding points in the images, the three-dimensional geometric information of the object is obtained. Therefore, binocular video can be considered as video captured simultaneously by two identical cameras from different viewpoints, which can acquire information about the object in three-dimensional space.

[0003] In related technologies, binocular video adds temporal information to the binocular image, making it susceptible to the influence of limited spatial resolution. Consequently, after super-resolution processing, it is prone to artifacts, occlusions, and other phenomena that result in poor video quality. Summary of the Invention

[0004] The purpose of this application is to provide a video processing method, apparatus, and electronic device to improve the artifacts and occlusions that exist in the video after super-resolution processing, thereby effectively improving the video quality.

[0005] In a first aspect, embodiments of this application provide a video processing method, comprising: acquiring videos to be processed captured from at least two perspectives; for any video frame to be processed, calculating the monocular similarity between corresponding features of the video frame to be processed and multiple adjacent video frames to be processed, and using the monocular similarity to supplement the feature information of the video frame to be processed to obtain supplementary features; for any two video frames to be processed captured at the same time point but from different perspectives, calculating the binocular similarity between the supplementary feature pairs corresponding to the two video frames to be processed based on prior knowledge, and fusing the supplementary feature pairs using the binocular similarity; wherein the prior knowledge includes prior knowledge of real-time localization and map construction; and performing super-resolution processing on the video to be processed based on the fused features to obtain a target video. In this way, refined fused features can be obtained based on the spatial information, temporal dimension information, and multi-view information within the video frame to be processed, thereby improving artifacts, occlusions, and other phenomena existing in the video after super-resolution processing, effectively improving video quality.

[0006] Optionally, calculating the binocular similarity between the supplementary feature pairs corresponding to the two video frames to be processed based on prior knowledge includes: for any supplementary feature pair, calculating the binocular similarity between any first supplementary feature and multiple second supplementary features; wherein the first supplementary feature is a supplementary feature corresponding to any of the video frames to be processed, and the second supplementary features are supplementary features corresponding to another video frame to be processed; the first supplementary feature and the second supplementary features are located on the same horizontal line. In this way, a binocular similarity map of dimension HWW can be calculated based on the video frames to be processed with dimension CHW. This omits the parameter calculation process on the channel dimension, effectively improving the computation speed.

[0007] Optionally, fusing the supplementary feature pairs using the binocular similarity includes: calculating the corresponding binocular weight for the binocular similarity of any supplementary feature pair using an exponential normalization function; weighting the binocular weight and the corresponding supplementary feature pair for any binocular weight; and using the multiple weighted results and the supplementary feature pairs as input to a convolutional neural network to fuse the supplementary feature pairs using the convolutional neural network. This allows the exponential normalization function to improve the generalization ability of this implementation process, making it applicable to more application scenarios.

[0008] Optionally, calculating the monocular similarity between corresponding features of any video frame to be processed and multiple adjacent video frames to be processed includes: calculating a first monocular similarity between the video frame to be processed and its own features using a convolutional neural network; and calculating a second monocular similarity between the features of the video frame to be processed and other adjacent video frames to be processed using a convolutional neural network. This allows for the separate calculation of the first and second monocular similarities, using the first monocular similarity to constrain the second monocular similarity, thus preventing the monocular weights corresponding to the monocular similarities from becoming too large in subsequent processes, which could reduce the quality of the video frames to be processed.

[0009] Optionally, the step of supplementing the feature information of the video frame to be processed using the monocular similarity to obtain supplementary features includes: calculating the corresponding monocular weights of the first monocular similarity and multiple second monocular similarities respectively using an exponential normalization function; for any monocular weight, weighting the monocular weight with the features of the video frame to be processed corresponding to the monocular weight; concatenating multiple weighted processing results into an array that meets the channel number requirement, and using the array as the input of a convolutional neural network so that the convolutional neural network outputs the supplementary features. In this way, the exponential normalization function can be used to improve the generalization ability of this implementation process, making it applicable to more application scenarios.

[0010] Optionally, the step of performing super-resolution processing on the video to be processed based on the fused features to obtain the target video includes: for any pixel corresponding to the fused features, extending the pixel into multiple pixels according to the number of channels corresponding to that pixel; wherein the number of channels is greater than 3; reconstructing the target video frame corresponding to the video frame to be processed using the extended multiple pixels, and then performing super-resolution processing on the target video frame to obtain the target video. In this way, by reducing the number of channels, a single pixel can be extended into multiple pixels, and a high-resolution target video can be reconstructed based on the extended multiple pixels. This eliminates the need for reconstruction operations using convolutional neural networks, improving the computational speed.

[0011] Secondly, embodiments of this application provide a video processing apparatus, comprising: an acquisition module for acquiring videos to be processed captured from at least two viewpoints; a supplementation module for calculating, for any video frame to be processed, the monocular similarity between corresponding features of the video frame to be processed and multiple adjacent video frames to be processed, and supplementing the feature information of the video frame to be processed using the monocular similarity to obtain supplementary features; a fusion module for calculating, based on prior knowledge, the binocular similarity between corresponding supplementary feature pairs of any two video frames to be processed captured at the same time point but from different viewpoints, and fusing the supplementary feature pairs using the binocular similarity; wherein the prior knowledge includes prior knowledge of real-time localization and map construction; and a processing module for performing super-resolution processing on the video to be processed based on the fused features to obtain a target video. In this way, refined fused features can be obtained based on the spatial information, temporal dimension information, and multi-viewpoint information within the video frame to be processed, thereby improving artifacts, occlusions, and other phenomena existing in the video after super-resolution processing, effectively improving video quality.

[0012] Thirdly, embodiments of this application provide an electronic device, including a processor and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps of the method provided in the first aspect above are performed.

[0013] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the steps of the method provided in the first aspect above.

[0014] Other features and advantages of this application will be set forth in the following description and will be apparent in part from the description or may be learned by practicing embodiments of this application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0015] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 A flowchart illustrating a video processing method provided in an embodiment of this application;

[0017] Figure 2 A structural block diagram of a video processing apparatus provided in an embodiment of this application;

[0018] Figure 3 This is a schematic diagram of the structure of an electronic device for performing a video processing method, provided as an embodiment of this application. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0020] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0021] It should be noted that, unless otherwise specified, the embodiments or technical features in the embodiments of this application may be combined.

[0022] In related technologies, after super-resolution processing, binocular videos are prone to artifacts and occlusions, resulting in poor video quality. To improve this, this application provides a video processing method, apparatus, and electronic device. Furthermore, by using the similarity between adjacent video frames captured from each viewpoint, feature information of each video frame to be processed is supplemented, and then the supplementary features between corresponding video frames from different viewpoints are fused. This provides more detailed information based on the temporal and multi-view dimensions, thereby improving artifacts and occlusions in the video after super-resolution processing and effectively enhancing video quality.

[0023] In some applications, the aforementioned video processing methods can be applied to medical image processing, where higher-quality video helps doctors more accurately determine the affected area. They can also be applied to video playback (e.g., in cinemas), where higher-quality video enhances the viewing experience. Furthermore, they can be applied to the transportation sector, where higher-quality video helps traffic management departments more accurately assess traffic conditions or assists vehicles in more precise obstacle avoidance. It should be noted that these video processing methods are not limited to the aforementioned application scenarios; they are applicable to other scenarios requiring high-quality super-resolution video, which will not be listed here.

[0024] In some application scenarios, the above video processing method can be applied to terminals or servers with video information processing capabilities. The terminals may include, for example, vehicle-mounted terminals, robot terminals, etc.; the servers may include, for example, independent servers or server clusters composed of multiple servers. Exemplarily, this application is hereinafter referred to as being applied to terminals.

[0025] The defects in the solutions in the above-mentioned related technologies are all the result of the inventors' practice and careful research. Therefore, the discovery process of the above problems and the solutions proposed by the embodiments of the present invention in the following text should be the inventors' contributions to the present invention.

[0026] Please refer to Figure 1 The diagram illustrates a flowchart of a video processing method provided in an embodiment of this application. Figure 1 As shown, the video processing method includes the following steps 101 to 104.

[0027] Step 101: Obtain the video to be processed from at least two perspectives;

[0028] In some application scenarios, a terminal can acquire videos to be processed from multiple perspectives. In these scenarios, multiple cameras can be installed on the terminal, and then these cameras at different locations can capture videos from different perspectives to obtain videos to be processed from multiple viewpoints.

[0029] In these application scenarios, the video to be processed may include stereo video. The terminal can then be equipped with a stereo camera to capture the corresponding stereo video. Once the stereo video recording is complete, the terminal obtains the video to be processed.

[0030] Step 102: For any video frame to be processed, calculate the monocular similarity between the corresponding features of the video frame to be processed and multiple adjacent video frames to be processed, and use the monocular similarity to supplement the feature information of the video frame to be processed to obtain supplementary features.

[0031] In some application scenarios, the terminal can process monocular videos captured from each viewpoint separately. Specifically, when processing any video frame, the terminal can extract features from each frame. In these scenarios, for example, convolutional neural networks can be used to extract features from the video frames. The extracted features can include, for example, color features, texture features, shape features, and other features obtained using the internal spatial information of the video frame.

[0032] After the terminal extracts the features of the video frame to be processed, it can calculate the similarity between the corresponding features of the video frame to be processed and other adjacent video frames to be processed. This similarity is the monocular similarity mentioned above. In some application scenarios, the monocular similarity can be calculated through contrastive learning, generative adversarial networks, etc.

[0033] After calculating the monocular similarity between any video frame to be processed and other video frames to be processed, the terminal can use this monocular similarity to supplement the feature information of the video frame to be processed. That is, the terminal can use the feature information corresponding to adjacent video frames to supplement the features of any video frame to be processed, so as to obtain more refined supplementary features. In some application scenarios, the terminal can, for example, set the weight of each similarity based on experience, and can perform weighted processing with the pixel values ​​corresponding to the original features of the video frame to be processed, thereby obtaining the above-mentioned supplementary features.

[0034] It should be noted that in the process of obtaining supplementary features by using monocular similarity between multiple video frames to be processed from the same viewpoint, since the shooting time of each video frame to be processed is different, this process can supplement the feature information in the time dimension (hereinafter referred to as time dimension information) of each video frame to be processed, thereby obtaining refined supplementary features.

[0035] Step 103: For any two video frames to be processed taken at the same time point but from different perspectives, calculate the binocular similarity between the supplementary feature pairs corresponding to the two video frames to be processed based on prior knowledge, and fuse the supplementary feature pairs using the binocular similarity; wherein, the prior knowledge includes prior knowledge of instant localization and map construction.

[0036] After the terminal obtains the supplementary features corresponding to each video frame to be processed, it can fuse any two video frames to be processed that were captured at the same time point but from different perspectives.

[0037] In some application scenarios, the terminal can, for example, fuse the left and right video frames to be processed from stereo video. Specifically, the terminal can calculate the stereo similarity between the supplementary feature pairs corresponding to the left and right video frames to be processed based on prior knowledge of Simultaneous Localization and Mapping (SLAM). This prior knowledge can be considered as follows: the two video frames to be processed should have been captured on the same horizontal line, thus the similarity (i.e., stereo similarity) between their corresponding supplementary feature pairs can be calculated to complement each other. The supplementary feature pairs here include identical supplementary features extracted from the two video frames to be processed. For example, the supplementary feature pair corresponding to video frames A and B to be processed includes feature a extracted from video frame A and feature b extracted from video frame B, where features a and b are obtained by supplementing feature information through monocular similarity. In these application scenarios, the above-mentioned stereo similarity can also be calculated through contrastive learning, generative adversarial networks, etc.

[0038] Then, the terminal can use binocular similarity fusion to supplement feature pairs, thereby adding features from other perspectives. In some application scenarios, the process of terminal fusing and supplementing feature pairs can be similar to the process of obtaining supplementary features described above, and will not be elaborated here.

[0039] It should be noted that the process of using the binocular similarity fusion between any two video frames to be processed from different perspectives to supplement feature pairs can be regarded as a process of supplementing each video frame to be processed with feature information from other perspectives (hereinafter referred to as multi-view information), thereby obtaining more comprehensive fused features. For example, for the left video frame to be processed in a binocular video, its fused features can include features that cannot be captured by the camera from the right perspective or the left perspective.

[0040] Step 104: Perform super-resolution processing on the video to be processed based on the fused features to obtain the target video.

[0041] After fusing the supplementary feature pairs, the terminal can perform super-resolution processing on the video to be processed based on the fused features to obtain the target video. In some application scenarios, the terminal can, for example, use a convolutional neural network to perform super-resolution processing on the video to be processed.

[0042] It should be noted that those skilled in the art, after obtaining the fused features, do not need to expend creative effort to reconstruct the high-resolution target video based on them, and this will not be elaborated here.

[0043] In this embodiment, the terminal can obtain refined fusion features based on the spatial information within the video frame to be processed, as well as the aforementioned temporal dimension information and multi-view information. This can improve the artifacts and occlusion phenomena that exist in the video after super-resolution processing, effectively improving the video quality.

[0044] Furthermore, the spatial, temporal, and multi-view information mentioned above are all processed decoupled. Therefore, if the video quality is not ideal, the cause of the poor video quality can be specifically identified from these three aspects. Moreover, by decoupling these three aspects, the parameter utilization rate in the calculation process can be improved, redundant parameters can be eliminated, thereby increasing the calculation speed and achieving the goal of improving processing efficiency.

[0045] In some optional implementations, the step 103 above, which calculates the binocular similarity between the supplementary feature pairs corresponding to the two video frames to be processed based on prior knowledge, includes: for any supplementary feature pair, calculating the binocular similarity between any first supplementary feature and multiple second supplementary features respectively; wherein, the first supplementary feature is a supplementary feature corresponding to any of the video frames to be processed, and the second supplementary feature is a supplementary feature corresponding to another video frame to be processed; the first supplementary feature and the second supplementary feature are located on the same horizontal line.

[0046] In some application scenarios, when a terminal calculates binocular similarity based on prior knowledge of SLAM, it can calculate the binocular similarity between each first supplementary feature located on the same horizontal line and multiple second supplementary features. For example, if the image dimensions of video frame A and video frame B to be processed are CHW (where C represents the number of channels; H represents the height; and W represents the width), then for multiple first supplementary features a1, a2, a3... a... located on the same horizontal line in video frame A to be processed... n This allows us to identify multiple second supplementary features b1, b2, b3...b in the video frame B to be processed, which are also located on the same horizontal line. n Here, 'n' represents the number of supplementary features. Then, we can calculate the relationship between a1 and b1, b2, b3...b1 respectively. n The similarity between them, a2 and b1, b2, b3...bn The similarity between them, a3 and b1, b2, b3...b n The similarity between them is calculated until a is obtained. n Each of the following is related to b1, b2, b3...b n The similarity between them. Thus, for two video frames to be processed with a height of H (which can be regarded as H horizontal lines), the dimension of the binocular similarity map calculated by the terminal is HWW.

[0047] In this implementation, a binocular similarity map with dimension HWW can be calculated based on the video frames to be processed, which have dimension CHW. This eliminates the need for parameter calculations along the channel dimensions, effectively improving the processing speed.

[0048] In some alternative implementations, the step 103 above, which involves fusing the supplementary feature pair using the binocular similarity, may include the following sub-steps:

[0049] Sub-step 1031: For any pair of supplementary features, calculate the corresponding binocular weights using the exponential normalization function to determine the binocular similarity.

[0050] In some application scenarios, the terminal can calculate the weights (i.e., binocular weights) corresponding to the binocular similarity using the exponential normalization function (Softmax function). For example, for video frame A and video frame B to be processed, after calculating the binocular similarity map with the above-mentioned HWW dimension, the corresponding binocular weights can be calculated for each binocular similarity using the Softmax function.

[0051] Sub-step 1032: For any binocular weight, perform weighted processing on the binocular weight and the supplementary feature pair corresponding to the binocular weight;

[0052] After the terminal calculates the binocular weights corresponding to each binocular similarity using the Softmax function, it can then perform a weighted average of each binocular weight and its corresponding supplementary feature. For example, for video frame A to be processed, each binocular weight is weighted averaged with its corresponding supplementary feature; then, for video frame B to be processed, the same weighted average is performed. In some application scenarios, the weighting process can be achieved by multiplying the pixel value corresponding to each supplementary feature with the binocular weight value.

[0053] Sub-step 1033: The multiple weighted processing results and the supplementary feature pairs are used as inputs to the convolutional neural network to fuse the supplementary feature pairs using the convolutional neural network.

[0054] In some application scenarios, the terminal can use each weighted processing result and its corresponding first and second supplementary features as input to a convolutional neural network (CNN). This CNN then supplements the first and second supplementary features with corresponding multi-view information, resulting in fused features. Alternatively, in these scenarios, the terminal can concatenate each weighted processing result and its corresponding first and second supplementary features into a new array using the `concat` method (used to connect two or more arrays) before inputting it into the CNN to maintain the correspondence between the three.

[0055] In this implementation, the exponential normalization function can be used to improve the generalization ability of the implementation process, so that it can be applied to more application scenarios.

[0056] It should be noted that this implementation process can be integrated into a model, and then the generalization ability of the model can be improved by the exponential normalization function, so that the model can be applied to more application scenarios.

[0057] In some optional implementations, the step 102 above, which calculates the monocular similarity between the corresponding features of any video frame to be processed and multiple adjacent video frames to be processed, includes: calculating the first monocular similarity between the video frame to be processed and its own features using a convolutional neural network for any video frame to be processed; and calculating the second monocular similarity between the features of the video frame to be processed and other adjacent video frames to be processed using a convolutional neural network.

[0058] In some application scenarios, when calculating monocular similarity, the terminal can use a convolutional neural network to calculate the first monocular similarity between each video frame to be processed and its own features. That is, the terminal can input the video frame to be processed into the convolutional neural network, and the convolutional neural network can output the first monocular similarity between it and itself.

[0059] Furthermore, the terminal can also input other adjacent video frames to be processed into the convolutional neural network, and make the convolutional neural network output the second monocular similarity between each adjacent video frame to be processed and the video frame to be processed.

[0060] For example, for the t-th video frame to be processed, the terminal can input the t-th video frame to be processed, the (t-2)-th video frame to be processed, and the (t+2)-th video frame to be processed into a convolutional neural network, so as to use the convolutional neural network to output one first monocular similarity and four second monocular similarities.

[0061] It should be noted that the terminal can input the video frame to be processed and other adjacent video frames to be processed separately, and output the first monocular similarity and the second monocular similarity respectively. The terminal can also input the video frame to be processed and other adjacent video frames to be processed simultaneously, and output the first monocular similarity and the second monocular similarity together; this is not a limitation. Furthermore, since the similarity of the video frame to be processed is calculated using a convolutional neural network together with other adjacent video frames to be processed, the value corresponding to the first monocular similarity will not be 1.

[0062] In this implementation, the terminal can calculate the first monocular similarity and the second monocular similarity separately, so as to use the first monocular similarity to constrain the second monocular similarity, thereby avoiding the monocular weight corresponding to the monocular similarity being too large in the subsequent process, which would reduce the quality of the video frames to be processed.

[0063] In some optional implementations, the step 102 above, which involves using the monocular similarity to supplement the feature information of the video frame to be processed to obtain supplementary features, may include the following sub-steps:

[0064] Sub-step 1021: Calculate the corresponding monocular weights of the first monocular similarity and the multiple second monocular similarities using an exponential normalization function.

[0065] In some application scenarios, after the terminal calculates the first monocular similarity and multiple second monocular similarities, it can calculate the corresponding monocular weights using an exponential normalization function. For example, for the t-th video frame to be processed, the terminal can use the exponential normalization function to calculate the corresponding 5 monocular weights for the t-th, (t-2), and (t+2)-th video frames to be processed.

[0066] Sub-step 1022: For any monocular weight, perform weighted processing on the monocular weight and the features of the video frame to be processed corresponding to the monocular weight.

[0067] After the terminal calculates multiple monocular weights, each monocular weight can be weighted and summed with its corresponding original feature. For example, for the t-th video frame to be processed, its corresponding monocular weight can be weighted and summed with its corresponding original feature; then for the (t-1)-th video frame to be processed, its corresponding monocular weight can be weighted and summed with its corresponding original feature; and then for the (t+1)-th video frame to be processed, its corresponding monocular weight can be weighted and summed with its corresponding original feature. In some application scenarios, the weighting process can be achieved by multiplying the pixel value corresponding to each original feature with the monocular weight value.

[0068] Sub-step 1023 involves concatenating multiple weighted processing results into an array that meets the channel number requirement, and using the array as the input to a convolutional neural network so that the convolutional neural network outputs the supplementary features.

[0069] After the terminal receives multiple weighted processing results, it can use the `concat` method to concatenate these results into a channel-dimensional array. That is, multiple weighted processing results can be concatenated into an array according to the number of channels, and this array can be used as input to a convolutional neural network to maintain the correspondence between the multiple weighted processing results and output supplementary features.

[0070] In this implementation, the exponential normalization function can be used to improve the generalization ability of the implementation process, so that it can be applied to more application scenarios.

[0071] It should be noted that this implementation process can also be integrated into a model, and then the generalization ability of the model can be improved by the exponential normalization function, so that the model can be applied to more application scenarios.

[0072] In some optional implementations, the super-resolution processing of the video to be processed based on the fused features to obtain the target video in step 104 above may include the following sub-steps:

[0073] Sub-step 1041: For any pixel corresponding to the fusion feature, extend the pixel into multiple pixels according to the number of channels corresponding to the pixel; wherein the number of channels is greater than 3.

[0074] In some application scenarios, after obtaining the fused features (i.e., the fused characteristics), the terminal can perform channel-to-space operations on them. Specifically, since a pixel is usually similar in color to its surrounding pixels, meaning its red, green, and blue channel values ​​are similar, the terminal can expand a single pixel into multiple pixels based on the number of channels. For example, if pixel 'a' has 12 channels, to satisfy the RGB color mode (meaning the video frame to be processed includes the red, green, and blue channels), one pixel can be expanded into four pixels.

[0075] Sub-step 1042 involves reconstructing the target video frame corresponding to the video frame to be processed using the extended multiple pixels, and then performing super-resolution processing on the target video frame to obtain the target video.

[0076] Then, the terminal can reconstruct the target video frame corresponding to the video frame to be processed by the extended multiple pixels, and then increase the resolution of the target video frame to obtain a high-resolution target video.

[0077] In related technologies, convolutional neural networks are typically used to reconstruct the video to be processed into a high-resolution target video. However, this process is relatively cumbersome and reduces the computational speed.

[0078] In this implementation, by reducing the number of channels, a single pixel can be expanded into multiple pixels, and a high-resolution target video can then be reconstructed based on these expanded pixels. This eliminates the need for reconstruction operations using convolutional neural networks, thus improving computational speed.

[0079] Those skilled in the art will understand that, in the methods described in the specific embodiments, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0080] Please refer to Figure 2 This diagram illustrates a structural block diagram of a video processing apparatus according to an embodiment of this application. The video processing apparatus may be a module, program segment, or code on an electronic device. It should be understood that this apparatus is similar to the one described above. Figure 1 The method implementation corresponds to this and can be executed. Figure 1 The specific functions of the device involved in the method embodiments can be found in the description above. To avoid repetition, detailed descriptions are omitted here.

[0081] Optionally, the video processing device includes an acquisition module 201, a supplementation module 202, a fusion module 203, and a processing module 204. The acquisition module 201 is used to acquire videos to be processed captured from at least two viewpoints. The supplementation module 202 is used to calculate the monocular similarity between corresponding features of any video frame to be processed and multiple adjacent video frames to be processed, and to supplement the feature information of the video frame to be processed using the monocular similarity to obtain supplementary features. The fusion module 203 is used to calculate the binocular similarity between corresponding supplementary feature pairs of any two video frames to be processed captured at the same time point but from different viewpoints, based on prior knowledge, and to fuse the supplementary feature pairs using the binocular similarity; wherein the prior knowledge includes prior knowledge of instantaneous localization and map construction. The processing module 204 is used to perform super-resolution processing on the video to be processed based on the fused features to obtain a target video.

[0082] Optionally, the fusion module 203 is further configured to: calculate the binocular similarity between any first supplementary feature and multiple second supplementary features for any supplementary feature pair; wherein, the first supplementary feature is a supplementary feature corresponding to any of the video frames to be processed, and the second supplementary feature is a supplementary feature corresponding to another video frame to be processed; the first supplementary feature and the second supplementary feature are located on the same horizontal line.

[0083] Optionally, the fusion module 203 is further configured to: calculate the corresponding binocular weight for the binocular similarity corresponding to any supplementary feature pair using an exponential normalization function; perform weighted processing on the binocular weight and the supplementary feature pair corresponding to the binocular weight for any binocular weight; and use the multiple weighted processing results and the supplementary feature pair as input to the convolutional neural network to fuse the supplementary feature pair using the convolutional neural network.

[0084] Optionally, the supplementary module 202 is further configured to: for any video frame to be processed, use a convolutional neural network to calculate a first monocular similarity between the video frame to be processed and its own features; and use a convolutional neural network to calculate a second monocular similarity between the features of the video frame to be processed and other adjacent video frames to be processed.

[0085] Optionally, the supplementary module 202 is further configured to: supplement the feature information of the video frame to be processed using the monocular similarity to obtain supplementary features, including: calculating the corresponding monocular weights of the first monocular similarity and multiple second monocular similarities respectively through an exponential normalization function; for any monocular weight, performing weighted processing on the monocular weight and the features of the video frame to be processed corresponding to the monocular weight; concatenating multiple weighted processing results into an array that meets the channel number requirement, and using the array as the input of a convolutional neural network so that the convolutional neural network outputs the supplementary features.

[0086] Optionally, the processing module 204 is further configured to: for any pixel corresponding to the fusion feature, extend the pixel into multiple pixels according to the number of channels corresponding to the pixel; wherein the number of channels is greater than 3; reconstruct the target video frame corresponding to the video frame to be processed through the extended multiple pixels, so as to perform super-resolution processing on the target video frame to obtain the target video.

[0087] It should be noted that those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system or device described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0088] Please refer to Figure 3 , Figure 3This is a schematic diagram of an electronic device for performing a video processing method, provided in an embodiment of this application. The electronic device may include: at least one processor 301, such as a CPU, at least one communication interface 302, at least one memory 303, and at least one communication bus 304. The communication bus 304 is used to enable direct communication between these components. In this embodiment, the communication interface 302 is used for signaling or data communication with other node devices. The memory 303 may be a high-speed RAM or a non-volatile memory, such as at least one disk storage device. Optionally, the memory 303 may also be at least one storage device located remotely from the aforementioned processor. The memory 303 stores computer-readable instructions. When these computer-readable instructions are executed by the processor 301, the electronic device can perform the aforementioned... Figure 1 The method and process are shown.

[0089] Understandable. Figure 3 The structure shown is for illustrative purposes only; the electronic device may also include components that are more advanced than those shown. Figure 3 The more or fewer components shown, or having the same Figure 3 The different configurations shown. Figure 3 The components shown can be implemented using hardware, software, or a combination thereof.

[0090] This application provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it can perform actions such as... Figure 1 The method process executed by the electronic device in the illustrated method embodiment.

[0091] This application provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, and when the program instructions are executed by a computer, the computer can execute the methods provided in the above-described method embodiments. For example, the method may include: acquiring videos to be processed captured from at least two viewpoints; for any video frame to be processed, calculating the monocular similarity between corresponding features of the video frame to be processed and multiple adjacent video frames to be processed, and using the monocular similarity to supplement the feature information of the video frame to be processed to obtain supplementary features; for any two video frames to be processed captured at the same time point but from different viewpoints, calculating the binocular similarity between the supplementary feature pairs corresponding to the two video frames to be processed based on prior knowledge, and fusing the supplementary feature pairs using the binocular similarity; wherein, the prior knowledge includes prior knowledge of instantaneous localization and map construction; and performing super-resolution processing on the video to be processed based on the fused features to obtain a target video.

[0092] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0093] Furthermore, the units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0094] Furthermore, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0095] In this document, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, without necessarily requiring or implying any such actual relationship or order between these entities or operations.

[0096] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A video processing method, characterized in that, include: Acquire videos to be processed from at least two different perspectives; For any video frame to be processed, a first monocular similarity between the video frame to be processed and its own features is calculated using a convolutional neural network, and a second monocular similarity between the features of the video frame to be processed and other adjacent video frames to be processed is calculated using a convolutional neural network. The first monocular similarity and the multiple second monocular similarities are respectively calculated using an exponential normalization function to obtain the corresponding monocular weights. For any monocular weight, the monocular weight is weighted and processed in conjunction with the features of the video frame to be processed corresponding to that monocular weight. Multiple weighted processing results are concatenated into an array that meets the channel number requirement, and the array is used as the input to a convolutional neural network so that the convolutional neural network outputs supplementary features. For any two video frames to be processed taken at the same time point but from different perspectives, based on prior knowledge, the binocular similarity between the supplementary feature pairs corresponding to the two video frames to be processed is calculated, and the supplementary feature pairs are fused using the binocular similarity; wherein, the prior knowledge includes prior knowledge of instant localization and map construction. Based on the fused features, super-resolution processing is performed on the video to be processed to obtain the target video; The step of calculating the binocular similarity between the supplementary feature pairs corresponding to the two video frames to be processed based on prior knowledge includes: For any pair of supplementary features, calculate the binocular similarity between any first supplementary feature and multiple second supplementary features; Wherein, the first supplementary feature is a supplementary feature corresponding to any of the video frames to be processed, and the second supplementary feature is a supplementary feature corresponding to another video frame to be processed; the first supplementary feature and the second supplementary feature are located on the same horizontal line; The process of fusing the supplementary feature pair using the binocular similarity includes: For any pair of supplementary features, the corresponding binocular similarity is used to calculate the corresponding binocular weight using an exponential normalization function. For any binocular weight, the weight is weighted and the corresponding supplementary feature pair is weighted. The multiple weighted processing results and the supplementary feature pairs are used as input to a convolutional neural network to fuse the supplementary feature pairs using the convolutional neural network.

2. The method according to claim 1, characterized in that, The step of performing super-resolution processing on the video to be processed based on the fused features to obtain the target video includes: For any pixel corresponding to the fusion feature, the pixel is extended into multiple pixels according to the number of channels corresponding to the pixel; wherein the number of channels is greater than 3. The target video frame corresponding to the video frame to be processed is reconstructed by extending multiple pixels, so as to perform super-resolution processing on the target video frame and obtain the target video.

3. A video processing apparatus, characterized in that, include: The acquisition module is used to acquire videos to be processed from at least two different perspectives. The supplementary module is used to calculate, for any video frame to be processed, a first monocular similarity between the video frame to be processed and its own features using a convolutional neural network, and a second monocular similarity between the features of the video frame to be processed and other adjacent video frames to be processed using a convolutional neural network; calculate the corresponding monocular weights of the first monocular similarity and multiple second monocular similarities using an exponential normalization function; for any monocular weight, perform weighted processing on the monocular weight and the features of the video frame to be processed corresponding to the monocular weight; concatenate multiple weighted processing results into an array that meets the channel number requirement, and use the array as the input of the convolutional neural network so that the convolutional neural network outputs supplementary features; The fusion module is used to calculate the binocular similarity between supplementary feature pairs corresponding to any two video frames to be processed, taken at the same time point but from different perspectives, based on prior knowledge, and to fuse the supplementary feature pairs using the binocular similarity; wherein, the prior knowledge includes prior knowledge of instant localization and map construction. The processing module is used to perform super-resolution processing on the video to be processed based on the fused features to obtain the target video; The step of calculating the binocular similarity between the supplementary feature pairs corresponding to the two video frames to be processed based on prior knowledge includes: For any pair of supplementary features, calculate the binocular similarity between any first supplementary feature and multiple second supplementary features; Wherein, the first supplementary feature is a supplementary feature corresponding to any of the video frames to be processed, and the second supplementary feature is a supplementary feature corresponding to another video frame to be processed; the first supplementary feature and the second supplementary feature are located on the same horizontal line; The process of fusing the supplementary feature pair using the binocular similarity includes: For any pair of supplementary features, the corresponding binocular similarity is used to calculate the corresponding binocular weight using an exponential normalization function. For any binocular weight, the weight is weighted and the corresponding supplementary feature pair is weighted. The multiple weighted processing results and the supplementary feature pairs are used as input to a convolutional neural network to fuse the supplementary feature pairs using the convolutional neural network.

4. An electronic device, characterized in that, It includes a processor and a memory, the memory storing computer-readable instructions that, when executed by the processor, perform the method as described in any one of claims 1-2.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it performs the method as described in any one of claims 1-2.

Images