Video processing method and device, electronic equipment, storage medium and chip

Abstract

The invention provides a video processing method and device, electronic equipment, a storage medium and a chip, and relates to the technical field of video processing. The method comprises the following steps: acquiring first depth information which is acquired by a terminal and corresponds to an image frame, and correcting the first depth information to obtain second depth information, so that the accuracy of the second depth information is improved, and the second depth information can reflect depth characteristics of a video scene more truly; and performing frame insertion processing on the first video in combination with the second depth information and the image frame to obtain the second video, thereby reducing the problem of frame insertion error or blurring caused by inaccurate depth information, and further improving the video processing effect.

Description

Technical Field

[0001] This disclosure relates to the field of video processing technology, and in particular to a video processing method, apparatus, electronic device, storage medium, and chip. Background Technology

[0002] A higher frame rate results in smoother video playback. By interpolating frames into a video, the frame rate can be increased, thereby improving playback smoothness.

[0003] In video processing technologies, models are typically used to predict the depth information corresponding to image frames in a video. This predicted depth information is then used to assist in frame interpolation. However, the accuracy of the depth information predicted by the model is low, resulting in poor video processing performance. Summary of the Invention

[0004] This disclosure provides a video processing method, apparatus, electronic device, storage medium, and chip to solve problems in the related art.

[0005] A first aspect of this disclosure provides a video processing method, the method comprising: Acquire image frames from the first video captured by the terminal, and acquire the first depth information corresponding to the image frames collected by the terminal; The first depth information is corrected to obtain the second depth information; Based on the image frames and the second depth information, the first video is subjected to frame interpolation to obtain the second video.

[0006] In some embodiments, the step of correcting the first depth information to obtain the second depth information includes: The first depth information is subjected to fuzzy filtering to obtain filtered depth information. The filtered depth information is adjusted at the edges to obtain the second depth information.

[0007] In some embodiments, performing edge adjustment on the filtered depth information to obtain the second depth information includes: Extract the first edge information from the filtered depth information, and extract the second edge information from the first depth information; Based on the difference between the first edge information and the second edge information, edge adjustment is performed on the filtered depth information to obtain the second depth information.

[0008] In some embodiments, adjusting the filtered depth information based on the difference between the first edge information and the second edge information to obtain the second depth information includes: If the difference is greater than a preset threshold, the first edge information in the filtered depth information is updated to the second edge information to obtain the second depth information; If the difference is less than or equal to the preset threshold, the filtered depth information is determined as the second depth information.

[0009] In some embodiments, the step of performing frame interpolation on the first video based on the image frame and the second depth information to obtain the second video includes: Based on the adjacent depth information of adjacent frames in the image frames of the first video in the second depth information, the first optical flow of the adjacent frames is generated; Based on the first optical flow and the adjacent frames, generate the second optical flow of the adjacent frames and the weights corresponding to the second optical flow; Based on the second optical flow and the weights, the first video is subjected to frame interpolation to obtain the second video.

[0010] In some embodiments, the step of performing frame interpolation on the first video based on the second optical flow and the weights to obtain the second video includes: Based on the second optical flow and the weight, an intermediate image frame between the adjacent frames is generated; Based on the intermediate image frames, the first video is subjected to the frame interpolation process to obtain the second video.

[0011] In some embodiments, generating intermediate image frames between adjacent frames based on the second optical flow and the weights includes: The second optical flow and the adjacent frames are subjected to distorted sampling processing to obtain the processed image frame; The intermediate image frame is generated based on the processed image frame and the weight.

[0012] A second aspect of this disclosure provides a video processing apparatus, comprising: The acquisition unit is used to acquire image frames from a first video captured by the terminal, and to acquire first depth information corresponding to the image frames collected by the terminal. The first processing unit is used to correct the first depth information to obtain the second depth information; The second processing unit is used to perform frame interpolation processing on the first video based on the image frame and the second depth information to obtain the second video.

[0013] In some embodiments, the first processing unit includes: The first processing module is used to perform fuzzy filtering on the first depth information to obtain filtered depth information. An adjustment module is used to perform edge adjustment on the filtered depth information to obtain the second depth information.

[0014] In some embodiments, the adjustment module is further configured to: Extract the first edge information from the filtered depth information, and extract the second edge information from the first depth information; Based on the difference between the first edge information and the second edge information, edge adjustment is performed on the filtered depth information to obtain the second depth information.

[0015] In some embodiments, the adjustment module is further configured to: If the difference is greater than a preset threshold, the first edge information in the filtered depth information is updated to the second edge information to obtain the second depth information; If the difference is less than or equal to the preset threshold, the filtered depth information is determined as the second depth information.

[0016] In some embodiments, the second processing unit includes: The generation module is used to generate the first optical flow of the adjacent frames based on the adjacent depth information of the adjacent frames in the second depth information in the image frames of the first video; The generation module is used to generate a second optical flow of the adjacent frame and a weight corresponding to the second optical flow based on the first optical flow and the adjacent frame; The second processing module is used to perform frame interpolation on the first video according to the second optical flow and the weight to obtain the second video.

[0017] In some embodiments, the second processing module is further configured to: Based on the second optical flow and the weight, an intermediate image frame between the adjacent frames is generated; Based on the intermediate image frames, the first video is subjected to the frame interpolation process to obtain the second video.

[0018] In some embodiments, the second processing module is further configured to: The second optical flow and the adjacent frames are subjected to distorted sampling processing to obtain the processed image frame; The intermediate image frame is generated based on the processed image frame and the weight.

[0019] A third aspect of this disclosure provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the methods described in the first aspect of this disclosure.

[0020] A fourth aspect of this disclosure provides a non-transitory computer-readable storage medium that, when instructions in the storage medium are executed by a processor of a mobile terminal, enables the mobile terminal to perform the methods described in the first aspect of this disclosure.

[0021] According to a fifth aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the methods described in the embodiments of the first aspect of this disclosure.

[0022] In summary, according to the video processing method proposed in this disclosure, the method includes acquiring first depth information corresponding to image frames collected by the terminal, correcting the first depth information to obtain second depth information, thereby improving the accuracy of the second depth information. The second depth information can more realistically reflect the depth characteristics of the video scene. The first video is then interpolated by combining the second depth information with the image frames to obtain the second video. This reduces the problem of interpolation errors or blurring caused by inaccurate depth information, thereby improving the effect of video processing.

[0023] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0024] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0025] Figure 1 A flowchart of a video processing method provided in this embodiment of the disclosure; Figure 2 A flowchart illustrating another video processing method provided in this disclosure embodiment; Figure 3 This is a video processing flowchart provided in an embodiment of the present disclosure; Figure 4 A flowchart illustrating a depth information correction process provided in an embodiment of this disclosure; Figure 5 A flowchart illustrating an optical flow processing method provided in an embodiment of this disclosure; Figure 6This is a schematic diagram of a neural network structure provided in an embodiment of the present disclosure; Figure 7 This is a schematic diagram of the structure of a video processing apparatus provided in an embodiment of the present disclosure; Figure 8 This is a schematic diagram of another video processing apparatus provided in an embodiment of the present disclosure; Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure; Figure 10 This is a schematic diagram of the structure of a chip provided in an embodiment of the present disclosure. Detailed Implementation

[0026] Some embodiments of this disclosure will be described in detail herein, examples of which are illustrated in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. Various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent upon understanding this disclosure. For example, the order of operations described herein is merely illustrative and is not limited to those orders set forth herein, but can be changed as will become apparent upon understanding this disclosure, except for operations that must be performed in a particular order. Furthermore, for clarity and brevity, descriptions of features known in the art may be omitted.

[0027] The embodiments described in the following examples of this disclosure are not representative of all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0028] A higher frame rate results in smoother video playback. By interpolating frames into a video, the frame rate can be increased, thereby improving playback smoothness.

[0029] In video processing technologies, models are typically used to predict the depth information corresponding to image frames in a video. This predicted depth information is then used to assist in frame interpolation. However, the accuracy of the depth information predicted by the model is low, resulting in poor video processing performance.

[0030] Therefore, in order to solve the problems existing in the related technologies, this disclosure proposes a video processing method. The method includes acquiring first depth information corresponding to image frames collected by the terminal, correcting the first depth information to obtain second depth information, thereby improving the accuracy of the second depth information. The second depth information can more realistically reflect the depth characteristics of the video scene. The first video is interpolated by combining the second depth information and the image frames to obtain the second video. This reduces the problem of interpolation errors or blurring caused by inaccurate depth information, thereby improving the effect of video processing.

[0031] This disclosure is not exhaustive, but merely illustrative of some embodiments, and is not intended to limit the scope of protection of this disclosure. Unless otherwise specified, each step in a particular embodiment can be implemented as an independent embodiment, and the steps can be arbitrarily combined. For example, a solution after removing some steps in a particular embodiment can also be implemented as an independent embodiment, and the order of the steps in a particular embodiment can be arbitrarily interchanged. Furthermore, the optional implementation methods in a particular embodiment can be arbitrarily combined; moreover, the embodiments can be arbitrarily combined, for example, some or all steps of different embodiments can be arbitrarily combined, and a particular embodiment can be arbitrarily combined with the optional implementation methods of other embodiments. In each of the disclosed embodiments, unless otherwise specified or in case of logical conflict, the terminology and / or descriptions of the embodiments are consistent and can be referenced by each other. Technical features in different embodiments can be combined to form new embodiments based on their inherent logical relationships. The terminology used in the embodiments of this disclosure is for the purpose of describing particular embodiments only and is not intended to limit the scope of this disclosure. In this disclosure, unless otherwise stated, elements expressed in the singular form, such as "a," "an," "the," "the," "the," "the," "the," "the," "this," etc., can mean "one and only one," or "one or more," "at least one," etc. For example, when using articles such as "a," "an," "the," etc. in translation, the noun following the article can be understood as either a singular or a plural expression. In some embodiments, the terms “in response to…”, “in response to determining…”, “in the case of…”, “when…”, “if…”, “if…”, etc., can be used interchangeably. In some embodiments, the terms “greater than,” “greater than or equal to,” “not less than,” “more than,” “more than or equal to,” “not less than,” “higher than,” “higher than or equal to,” “not lower than,” and “above” can be used interchangeably, as can the terms “less than,” “less than or equal to,” “not greater than,” “less than,” “less than or equal to,” “not more than,” “lower than,” “lower than or equal to,” “not higher than,” and “below”. The prefixes such as "first" and "second" in the embodiments of this disclosure are only for distinguishing different descriptive objects and do not constitute restrictions on the position, order, priority, number or content of the descriptive objects. For the description of the descriptive objects, please refer to the description in the claims or the context of the embodiments. The use of prefixes should not constitute unnecessary restrictions.

[0032] In the embodiments disclosed herein, "multiple" refers to two or more. In the embodiments disclosed herein, terms such as “import”, “input”, and “read in” can be used interchangeably.

[0033] In some embodiments, devices, etc., can be interpreted as physical or virtual, and their names are not limited to the names recorded in the embodiments. Terms such as “device”, “equipment”, “circuit”, “network element”, “node”, “function”, “unit”, “section”, “system”, “network”, “chip”, “chip system”, “entity”, and “subject” can be used interchangeably.

[0034] Figure 1 This is a flowchart illustrating a video processing method provided in an embodiment of this disclosure. This method can be applied to application scenarios such as smart terminals, for example, executed by a terminal with integrated video processing functions or a video processor within a terminal, or executed by other devices suitable for performing video processing and outputting video processing results; this disclosure does not limit its application. Figure 1 As shown, the video processing method includes steps S101-S103.

[0035] Step S101: Obtain an image frame from the first video captured by the terminal, and obtain the first depth information corresponding to the image frame collected by the terminal.

[0036] In the embodiments of this disclosure, a terminal refers to an electronic device with video shooting and depth information acquisition capabilities, including but not limited to smartphones, tablets, cameras, and augmented reality (AR) / virtual reality (VR) devices; its hardware must integrate at least an image acquisition module (such as a red-green-blue (RGB) camera) and a depth acquisition module (such as a time-of-flight (TOF) sensor, a binocular stereo vision sensor, or a structured light sensor), and support the collaborative operation of the image acquisition module and the depth acquisition module. First video: refers to the raw video data directly captured by the terminal through the image acquisition module (without subsequent processing such as frame interpolation or depth correction), the content of which can cover everyday scenes (such as human movement, landscape recording), professional scenes (such as live sports footage, industrial monitoring footage), etc. Image frame: refers to a static image unit constituting the first video, which is a visual snapshot of the first video at a specific moment. First depth information: refers to a depth data map that is temporally aligned and spatially registered with each frame in the image frame. The first depth information refers to the raw depth data that the terminal directly collects through the depth acquisition module and that matches a certain image frame in the first video in both time and space, rather than being generated through model prediction. Its essence is to describe the real physical space distance corresponding to each pixel in the image frame (that is, the distance between the real object corresponding to the pixel and the terminal lens, usually in millimeters or centimeters). For example, the first depth information corresponding to the pixel of a person's face in the image frame is 50cm, and the first depth information corresponding to the pixel of the background wall is 300cm.

[0037] In one embodiment of this disclosure, the method is executed by a processing chip, which can capture a first video by calling a camera. Simultaneously, the processing chip can call a depth camera to obtain depth information corresponding to video frames / image frames in the first video.

[0038] In embodiments of this disclosure, the terminal captures video at a predetermined frame rate (e.g., 30 frames / second or 60 frames / second) using an image acquisition module to obtain a first video. The first video is stored in the terminal in compressed or uncompressed form. To extract image frames, the terminal's processing module (e.g., a video decoder or a dedicated image processing chip) segments the video data frame by frame, generating independent image frame sequences. Simultaneously, the terminal acquires depth information corresponding to the image frames. Depth information acquisition can be achieved in several ways: Binocular vision system: The terminal device uses two cameras to capture the same scene from different angles, and obtains depth information through parallax calculation. Time-of-Flight (ToF) technology: The terminal device uses an infrared laser emitter to emit light into the scene and measures the round-trip time of the light through a receiver to calculate the distance value of each pixel in the scene. Monocular depth estimation: The terminal device uses a single camera to capture an image, analyzes the image using a deep learning model (e.g., a convolutional neural network), and predicts depth information. Depth information is typically stored in the form of a depth map, which has the same resolution as the image frame, with each pixel storing a depth value. The terminal's processing module pairs the acquired image frames with their corresponding depth information, ensuring strict temporal synchronization between each image frame and its corresponding depth map. This association can be achieved through timestamps or frame numbers. The processing module stores the image frames and depth information in memory or a storage device for subsequent processing.

[0039] By simultaneously acquiring image frames and depth information, multimodal data support is provided for video processing. Image frames provide two-dimensional visual information of the scene, while depth information supplements the scene with three-dimensional spatial information, enabling subsequent frame interpolation processing to more accurately determine the direction and speed of object motion.

[0040] Step S102: Correct the first depth information to obtain the second depth information.

[0041] In the embodiments of this disclosure, the correction processing refers to the optimization operation performed to address the deficiencies of the first depth information. The core objective is to remove noise interference and enhance edge sharpness while retaining effective depth information (such as the actual distance of objects), making the processed depth information closer to the spatial structure of the real scene. The second depth information refers to the optimized depth information obtained after the correction processing. Its noise level is significantly reduced, the object edge contours are clearer, and the accuracy and stability of the depth value are better than the first depth information, which can be directly used for subsequent video frame interpolation calculations.

[0042] In the embodiments of this disclosure, the obtained first depth information undergoes correction processing. The main goal of the correction processing is to remove noise and erroneous values ​​from the depth information and optimize the continuity and consistency of the depth data, thereby obtaining more accurate second depth information. Specific correction processing methods may include the following aspects: Fuzzy filtering: The first depth information is fuzz-filtered to reduce depth value fluctuations caused by sensor errors or noise. Fuzzy filtering can smooth outliers in the depth map, making the depth values ​​smoother and more uniform. This process helps eliminate irregular fluctuations within a small range, especially in areas with large depth value changes. Edge adjustment: After fuzzing, the edge regions in the depth information are adjusted to ensure that the depth information maintains relatively accurate changes at the edges. Edge regions typically contain complex depth changes, and direct fuzzing may result in unclear edges; therefore, special processing of the edges is required. Edge data in the first depth information is extracted using an edge detection algorithm, and then adjusted according to the differences in edge information, making the transition of the depth map at the edges smoother and more accurate. Error value correction: For obvious depth value errors (such as depth errors caused by sensor errors, reflections, or occlusion), context information can be used for correction. Specifically, abnormal depth values ​​can be inferred and replaced using the depth values ​​of adjacent pixels, avoiding distortion of the entire depth map due to the influence of a single pixel error. After the above processing, the resulting depth map is the optimized second depth information. At this point, the depth data in the second depth information is more accurate and stable, better matches image frames, and provides more reliable spatial information for subsequent frame interpolation. The accuracy and consistency of the second depth information are significantly improved compared to the original first depth information, especially in complex scenes or areas with drastic depth changes, where the corrected second depth information exhibits higher accuracy and stability.

[0043] The correction process significantly improves the reliability and accuracy of depth information through operations such as denoising, calibration, and detail filling, making it closer to the depth distribution of the actual scene.

[0044] Step S103: Based on the image frame and the second depth information, perform frame interpolation on the first video to obtain the second video.

[0045] In the embodiments of this disclosure, frame interpolation is a video processing technique designed to increase the frame rate of a video by inserting additional intermediate frames between the original video frames, resulting in smoother and more natural video playback. Frame interpolation typically predicts the content of the scene at intermediate moments based on the image content and depth information of adjacent frames. The second video refers to the video obtained after frame interpolation. Compared to the first video, the second video has a higher frame rate, smoother and more natural scene transitions, and a better visual experience.

[0046] In the embodiments of this disclosure, image frames in the first video are analyzed frame by frame to extract key visual features such as color, brightness, edges, and texture. These features serve as important references for subsequent frame interpolation algorithms, helping to determine the image content of intermediate frames. Using second depth information, the optical flow field between adjacent image frames is calculated. The optical flow field describes the trajectory and velocity of pixels between consecutive frames. Combining this with depth information allows for a more accurate estimation of the three-dimensional motion state of pixels, including complex movements such as forward / backward movement, left / right translation, and rotation. Based on the calculated optical flow field and depth information, appropriate interpolation algorithms (such as bilinear interpolation, bicubic interpolation, etc.) are used to generate intermediate frames. During the interpolation process, the movement direction, velocity, and the preceding / following relationships indicated by the depth information of the pixels are fully considered to ensure that the generated intermediate frames visually transition naturally from the original frames and conform to the actual motion patterns of objects. The generated intermediate frames are inserted into the original video frame sequence in chronological order to form a new video frame sequence. During the insertion process, the weight distribution between the intermediate frames and the original frames is carefully adjusted to maintain visual continuity and consistency in the video, avoiding abnormal phenomena such as flickering and jitter. The generated second video undergoes further optimization, such as color correction, contrast enhancement, and sharpening, to improve the overall visual quality of the video and make it more suitable for users' viewing needs.

[0047] By inserting intermediate frames between the original video frames, the video frame rate is effectively increased, making video playback smoother and more natural, reducing stuttering and ghosting caused by low frame rates, and improving the user's viewing experience.

[0048] In summary, in the embodiments of this disclosure, the video processing method proposed in this disclosure includes acquiring first depth information corresponding to image frames collected by the terminal, correcting the first depth information to obtain second depth information, thereby improving the accuracy of the second depth information. The second depth information can more realistically reflect the depth features of the video scene. The first video is interpolated by combining the second depth information and the image frames to obtain the second video, thereby reducing the problem of interpolation errors or blurring caused by inaccurate depth information, and thus improving the effect of video processing.

[0049] As a refinement of step S102, when performing the correction processing on the first depth information to obtain the second depth information, it can be implemented in the following ways, but not limited to: performing fuzzy filtering processing on the first depth information to obtain filtered depth information; and performing edge adjustment on the filtered depth information to obtain the second depth information.

[0050] In the embodiments of this disclosure, fuzzy filtering refers to a signal processing technique aimed at suppressing random noise in data, smoothing discontinuous transition points, and filling small-scale data holes. Its core objective is to perform a weighted average of depth values ​​in the spatial domain, sacrificing local sharpness in exchange for overall data consistency and stability, generating depth data with smoother transitions. Filtered depth information refers to the intermediate result obtained after fuzzy filtering of the first depth information. Its characteristic is that in physically continuous flat areas (such as walls or tabletops), the depth values ​​become uniform, and noise is effectively suppressed; however, in depth discontinuous areas representing the geometric boundaries of objects, the edges become blurred and diffused due to the smoothing effect. Edge adjustment refers to a fine-tuning process aimed at restoring and sharpening the geometric boundaries of objects in the filtered depth information. This process corrects and enhances boundaries distorted by over-smoothing by introducing edge cues from the original data, thereby making the object's outline clearer and more accurate. The second depth information refers to the final optimized depth map after the two-stage filtering and adjustment processing. The second depth information retains the advantages of low noise and high consistency in uniform regions of the filtered depth information, while also possessing clear and sharp object boundaries, making it an ideal geometric prior information for high-quality video processing.

[0051] In embodiments of this disclosure, the first depth information is subjected to Gaussian blur filtering. Gaussian blurring smooths the depth information by calculating the weighted average of each pixel in the depth map and its neighboring pixels. The weights are assigned according to the distance between pixels, with closer pixels having larger weights. Edge detection is performed on the filtered depth information to extract edge information. Edge detection algorithms such as the Sobel operator or the Canny operator can be used to calculate the gradient changes of pixel values ​​in the filtered depth information and determine the edge positions of the filtered depth information. Based on the edge detection results, the edge regions of the filtered depth information are sharpened. Edge enhancement algorithms, such as the Laplacian operator or an adaptive filter, can be used to enhance the depth value changes of edge pixels in the filtered depth information. The edge information of the first depth information is used as a reference to correct the edges of the filtered depth information. By comparing the edge positions and depth value differences between the first depth information and the filtered depth information, the edges of the filtered depth information are adjusted to be closer to the edge features of the original depth information. The blurred and smoothed depth information is then fused with the enhanced and corrected edge information. Weighted fusion or region replacement methods can be used to embed enhanced edge information into filtered depth information to generate the final second depth information.

[0052] Blur filtering effectively reduces noise and artifacts in the initial depth information, making the depth information smoother and more continuous. Edge adjustment enhances the edge sharpness and accuracy of the depth information, restoring details lost due to blur filtering.

[0053] As a refinement of the above embodiments, when performing edge adjustment on the filtered depth information to obtain the second depth information, it can be implemented in the following ways, but not limited to: extracting first edge information from the filtered depth information and extracting second edge information from the first depth information; and performing edge adjustment on the filtered depth information according to the difference between the first edge information and the second edge information to obtain the second depth information.

[0054] In the embodiments of this disclosure, the first edge information refers to the edge features extracted from the filtered depth information, reflecting the contour boundary of the object in the depth map after noise reduction. It manifests as a set of pixels where the depth value changes significantly (e.g., a pixel transition area abruptly changes from 1m to 3m), but may experience edge position shifts or expanded transition ranges (blurring) due to filtering. The second edge information refers to the edge features extracted from the first depth information (the original unfiltered depth data), reflecting the object's contour boundary before noise reduction. It may contain a small amount of noise interference, but the edge position and the original depth abrupt change features are closer to the real scene (e.g., the actual physical boundary of the object). The difference in edge information refers to the difference between the first and second edge information, mainly including two categories: positional difference: the pixel coordinate shift of the same object edge in the two depth maps (e.g., the first edge is located at x=50 pixels, and the second edge is located at x=53 pixels); intensity difference: the abrupt change in depth value at the edge is different (e.g., the depth change of the first edge is 1m→2.5m, and the depth change of the second edge is 1m→3m). Edge adjustment refers to the operation of correcting edge regions in the filtered depth information based on the difference between the first edge information and the second edge information. The core principle is to restore or optimize the position and intensity of edges while preserving the filtering and noise reduction effect, making the edge features closer to the real scene.

[0055] In the embodiments of this disclosure, the core parameters for preset edge detection are: depth change threshold: when the depth value difference between adjacent pixels exceeds this threshold (e.g., 0.5m), it is determined to be a possible edge pixel; edge connection threshold: edge pixels with a distance ≤ 2 pixels and consistent depth change trends are connected into continuous edge lines to avoid interference from isolated edge points. Edge detection is performed on the filtered depth information (depth map after fuzzy filtering): the depth value difference with the adjacent 8 directional pixels is calculated pixel by pixel, and pixels with differences exceeding the depth change threshold are selected; connectivity analysis is performed on the selected pixels, and discrete pixels are merged into continuous edge lines through the edge connection threshold to form the first edge information (including the pixel coordinates of the edge, the magnitude of the depth change, etc.). The same edge detection process as above is performed on the first depth information (original depth map): based on the depth change threshold and the edge connection threshold, edge pixels in the original depth map are extracted and connected into continuous edge lines; since the first depth information may contain noise, false edge removal is required: isolated edge lines with a length less than a preset length threshold (which are likely false edges caused by noise) are removed, and finally the second edge information is formed. The core of difference assessment is to quantify the differences between the two types of edge information, providing a basis for subsequent adjustments: Edge matching and alignment: Spatial matching of the first and second edge information is performed using terminal algorithms: Taking the edge line of the second edge information as a benchmark, edge lines in the first edge information with a distance less than a preset distance threshold and consistent depth change direction (e.g., both from near to far) are searched, and a one-to-one matching relationship is established; For unmatched edge lines (e.g., edges that exist in the first edge information but not in the second edge information), they are marked as false edges introduced by filtering; For edges that exist in the second edge information but not in the first edge information, they are marked as lost edges caused by excessive filtering. Difference quantification calculation: For successfully matched edge lines, two types of difference indicators are calculated: Position difference: The average coordinate offset of all corresponding pixels on the edge line is taken (e.g., an average offset of 3 pixels in 100 pixels of a certain edge line); Intensity difference: The average difference of the depth change amplitude at the edge line is taken (e.g., the average change of the first edge is 1.5m, the average change of the second edge is 2.0m, and the intensity difference is 0.5m). Difference threshold setting: Preset difference judgment thresholds based on the terminal application scenario (can be dynamically adjusted): Position difference threshold: can be set to 3 pixels (exceeding this value indicates significant edge position deviation); Intensity difference threshold: can be set to 0.3m (exceeding this value indicates distortion of edge depth abrupt change characteristics). However, it should be clarified that this statement is not intended to limit the specific value of any of the above thresholds.

[0056] By adjusting the edges, issues such as blurred or misaligned edges are avoided. The processed second depth information more accurately reflects the true spatial distribution of objects and scenes, especially at object edges, contours, and transition areas, resulting in significantly improved accuracy.

[0057] As a refinement of the above embodiments, when performing edge adjustment on the filtered depth information based on the difference between the first edge information and the second edge information to obtain the second depth information, it can be implemented in the following ways, but not limited to: when the difference is greater than a preset threshold, updating the first edge information in the filtered depth information to the second edge information to obtain the second depth information; when the difference is less than or equal to the preset threshold, determining the filtered depth information as the second depth information.

[0058] In the embodiments of this disclosure, the preset threshold refers to a pre-set judgment criterion used to distinguish whether the differences in edge information are significant. The preset threshold can be adjusted according to the actual application scenario and the characteristics of the depth information, and is usually determined through experiments or experience.

[0059] In the embodiments of this disclosure, the first edge information and the second edge information are compared pixel by pixel to calculate their differences in position, intensity, or shape. The differences can be quantified by calculating the absolute value of the difference in gradient values, the offset of the edge position, or the difference in edge continuity. A reasonable preset threshold is set based on the actual application scenario and the characteristics of the depth information. For example, the threshold can be determined experimentally or empirically based on the noise level of the depth information and the edge detail requirements. If the difference between the first edge information and the second edge information exceeds the preset threshold, it indicates that the filtering process has caused the edge information to be lost or distorted. In this case, the first edge information in the filtered depth information is updated to the second edge information to restore the edge details. If the difference between the first edge information and the second edge information is within the preset threshold range, it indicates that the edge information after filtering is accurate enough. In this case, the filtered depth information is directly determined as the second depth information without further adjustment. In cases of significant differences, the second edge information is embedded into the filtered depth information, replacing the first edge information. This can be done by direct overlay or by weighted fusion to combine the second edge information with the filtered depth information to generate higher-quality second depth information. In cases of insignificant differences, the filtered depth information is directly used as the second depth information. This step ensures the smoothness and reliability of the depth information.

[0060] In cases of significant differences, edge details lost due to filtering are recovered by updating edge information, making the second depth information closer to the depth distribution of the actual scene.

[0061] When the differences are not significant, the filtered depth information is used directly, avoiding unnecessary adjustments and ensuring the smoothness and reliability of the depth information.

[0062] In one embodiment of this disclosure, the electronic device may also acquire third depth information of each image frame in the first video frame, wherein the third depth information is obtained by prediction. The process of interpolating the first video based on the image frames and the second depth information to obtain the second video includes: interpolating the first video based on the image quality in the first video frames and based on the first depth information and the second depth information to obtain the second video.

[0063] In this embodiment of the disclosure, image frame interpolation can be flexibly performed based on the depth information acquired by the depth sensor and the predicted depth information, according to the image quality. For example, when the average image quality in the first video is high, the proportion of frame interpolation using the third depth information is increased, thereby reducing power consumption. When the average image quality in the first video is low, the proportion of frame interpolation using the second depth information is increased, thereby improving the quality of the interpolated image.

[0064] Whether the quality known in this disclosure is low or high can be determined based on existing quality evaluation methods, and no limitation is made in this regard.

[0065] Figure 2 A flowchart of a video processing method proposed in this disclosure is further shown. Based on Figure 1 The illustrated embodiment further explains step S103. Figure 2 This may include the following steps: Step S201: Generate the first optical flow of the adjacent frames based on the adjacent depth information of the adjacent frames in the second depth information in the image frames of the first video.

[0066] In the embodiments of this disclosure, adjacent frames refer to two temporally consecutive frames in the first video, denoted as frame t and frame t+1 (frame t is the previous frame, and frame t+1 is the next frame). The time interval between them is determined by the frame rate of the first video (e.g., approximately 33ms in a 30fps video). Adjacent frames contain visual information of the same scene, but pixel positions may change due to object movement or terminal movement. Adjacent depth information refers to the second depth information corresponding to adjacent frames (frame t and frame t+1), denoted as depth t and depth t+1. Depth t is the corrected depth data for frame t, and depth t+1 is the corrected depth data for frame t+1. Both have low noise and clear edges, and correspond one-to-one with the pixels of the corresponding image frames. The first optical flow refers to the vector field describing the motion state of each pixel in adjacent frames (from frame t to frame t+1). Each vector contains a "direction of motion" (e.g., horizontal to the right, diagonally at 45 degrees) and a "distance of motion" (in pixels, e.g., 5 pixels), reflecting the spatial displacement of pixels from frame t to frame t+1. For example, when a person in the foreground moves, the optical flow vector corresponding to their pixel points in the direction of motion, while the optical flow vector of a stationary object in the background is close to zero.

[0067] In the embodiments of this disclosure, within each motion region, an initial motion vector is calculated using a pixel matching algorithm (by comparing the similarity of the grayscale values ​​of pixels in frame t and frame t+1) to obtain the original optical flow within the region. For example, in a foreground person region, the motion direction and distance of a pixel are determined by searching for the position in frame t+1 that is closest to the grayscale value of a pixel in frame t. The original optical flow is corrected using adjacent depth information: For foreground regions (small depth values): if the motion distance of the original optical flow is too small (not consistent with the predicted motion trend, such as a person moving quickly but the optical flow only moving 1 pixel), the search range is expanded and recalculated to ensure that the motion distance matches the depth; For background regions (large depth values): if the motion distance of the original optical flow is too large (such as the background building's optical flow moving 10 pixels), it is corrected by the average optical flow of adjacent pixels within the region to suppress outliers; For region edges (such as the boundary between a person and the background): combining the clear edges of the second depth information, the optical flow of edge pixels is restricted from crossing the region boundary (avoiding the incorrect attribution of the motion of foreground pixels to the background). The optical flow vectors within a region are smoothed. If the difference between the optical flow of a pixel and its 8 neighboring pixels exceeds a preset range (e.g., directional deviation > 30 degrees or distance difference > 3 pixels), the median of the neighboring optical flow is used to replace the optical flow of that pixel, ensuring that the optical flow movement trend is consistent within the same region. All optimized optical flow vectors from different regions are integrated to form the first optical flow covering the entire image frame. Each pixel corresponds to a precise motion vector, reflecting the true motion trajectory from frame t to frame t+1.

[0068] By utilizing second depth information, the motion of objects in three-dimensional space can be estimated more accurately. Especially when dealing with scenes with depth variations, it can better capture the true motion state of objects compared to optical flow calculation methods based solely on two-dimensional images.

[0069] Step S202: Based on the first optical flow and the adjacent frames, generate the second optical flow of the adjacent frames and the weights corresponding to the second optical flow.

[0070] In the embodiments of this disclosure, the second optical flow refers to a denser and more accurate motion vector field obtained by optimizing and refining the first optical flow. The second optical flow not only matches the first optical flow in macroscopic motion but also captures local deformations and subtle movements of the object's surface at the microscopic level. The weight refers to a confidence map that quantifies the reliability of each motion vector in the second optical flow, corresponding one-to-one. A higher weight value indicates a more reliable motion estimate at that location; a lower value indicates higher uncertainty in the motion estimate at that location, which may require careful handling in subsequent synthesis.

[0071] In the embodiments of this disclosure, a second optical flow is generated based on a first optical flow and adjacent frames, through further optimization and adjustment of the first optical flow. Specifically, motion information in adjacent frames is first analyzed, and the first optical flow is improved by combining the optical flow data of adjacent frames. By considering the continuity of time between frames, the characteristics of object motion, and the influence of depth information, the estimation result of the first optical flow is adjusted to obtain the second optical flow. Based on the generated second optical flow, the weight of each pixel is calculated. The calculation of the weight value depends not only on the amplitude of pixel motion but also on the stability of image content, changes in depth information, and the motion pattern of objects. For example, pixels with greater motion have higher weights because the motion changes of these pixels are more obvious and contribute more to the optical flow estimation. Pixels with smaller changes or blurriness have relatively lower weights. By continuously adjusting and optimizing the second optical flow, the accuracy of the overall optical flow field can be improved. This process usually combines multiple factors such as the distribution of objects in the scene, motion speed, and depth information for dynamic adjustment, so that the second optical flow is more consistent with the laws of object motion in the real world.

[0072] By generating a second optical flow, the accuracy of object motion estimation in video analytics can be improved. Compared to the first optical flow, the second optical flow takes into account more dynamic factors, effectively reducing motion blur and estimation errors.

[0073] Step S203: Based on the second optical flow and the weight, perform frame interpolation on the first video to obtain the second video.

[0074] In the embodiments of this disclosure, frame interpolation refers to increasing the frame rate of the video by inserting additional intermediate frames between the original video frames, making the video playback smoother and more natural.

[0075] In embodiments of this disclosure, the motion trajectory of each pixel between adjacent frames is predicted based on a second optical flow. The second optical flow provides the direction and velocity of pixel motion, accurately describing the displacement of pixels over time. The contribution of each pixel to the interpolation is adjusted according to weights. High-weight pixels (such as regions with high motion consistency) contribute more to the interpolation, while low-weight pixels (such as regions with unreliable depth information or motion blur) contribute less. For each pixel between adjacent frames, its position at the intermediate time step is predicted based on the second optical flow, and interpolation is performed in conjunction with the weights. The interpolation method can be bilinear interpolation or a more complex interpolation algorithm to generate pixel values ​​for the intermediate frame. An intermediate frame is generated based on the interpolation result and inserted into the original video frame sequence. After inserting the intermediate frame, a new video frame sequence, i.e., the second video, is formed. The generated second video is smoothed to eliminate artifacts or unnatural transitions that may occur during the interpolation process. The smoothing process can employ temporal domain filtering or spatial domain filtering methods to further improve the visual quality of the video.

[0076] Frame interpolation significantly improved the frame rate of the first video, resulting in smoother and more natural playback. Frame interpolation reduced stuttering and ghosting issues caused by low frame rates, enhancing the user's viewing experience.

[0077] As a refinement of step S203, when performing the frame interpolation process on the first video according to the second optical flow and the weight to obtain the second video, it can be implemented in the following ways, but is not limited to: generating an intermediate image frame between the adjacent frames according to the second optical flow and the weight; and performing the frame interpolation process on the first video according to the intermediate image frame to obtain the second video.

[0078] In the embodiments of this disclosure, an intermediate image frame refers to a transition frame generated between adjacent original frames (denoted as frame t and frame t+1) of the first video, and its time position is between the two original frames (e.g., when the interval between frame t and frame t+1 is 33ms, the intermediate frame is located at 16.5ms). Its core function is to fill the time gap between the original frames, making the transition of the picture more smooth during video playback and avoiding stuttering.

[0079] In embodiments of this disclosure, the motion vector of each pixel is further refined and optimized based on the second optical flow. This step aims to improve the accuracy of the optical flow field to ensure that the generated intermediate image frames are more accurate. For each pixel between adjacent frames, its position at the intermediate moment is predicted based on the second optical flow. The value of the pixel at the intermediate moment is calculated using an interpolation algorithm (such as bilinear interpolation or bicubic interpolation). The interpolation results are weighted and fused, considering the weight of each pixel. Pixels with high weight (such as regions with high motion consistency) contribute more to the interpolation results, while pixels with low weight (such as regions with unreliable depth information or motion blur) contribute less. Intermediate image frames are generated based on the above interpolation and fusion steps. The intermediate image frames are located between the original adjacent frames and reflect the scene content at the intermediate moment. The generated intermediate image frames are inserted into the adjacent frames of the first video in chronological order to form a new video frame sequence. The generated second video is smoothed to eliminate artifacts or unnatural transitions that may occur during the interpolation process. Smoothing can be achieved using time-domain or spatial-domain filtering methods to further improve the visual quality of the video.

[0080] By inserting intermediate image frames between the original video frames, the video frame rate is significantly increased, making video playback smoother and more natural. This reduces stuttering and ghosting issues caused by low frame rates, enhancing the user's viewing experience.

[0081] As a refinement of the above embodiments, when performing the step of generating an intermediate image frame of the adjacent frames based on the second optical flow and the weights, it can be implemented in the following manner, but is not limited to: performing distortion sampling processing on the second optical flow and the adjacent frames to obtain a processed image frame; and generating the intermediate image frame based on the processed image frame and the weights.

[0082] In embodiments of this disclosure, warp sampling refers to an image transformation technique that uses a motion vector field (second optical flow) to reposition the original image pixels to their spatial locations at a target time. This process generates a predicted view of the scene content at a new time point by resampling pixels along a motion trajectory. The processed image frame refers to the intermediate result image obtained through warp sampling, after motion compensation of the original frame content at the target time.

[0083] In embodiments of this disclosure, adjacent frames are subjected to distorted sampling processing based on a second optical flow. The specific steps are as follows: For each pixel in an adjacent frame, its position at the intermediate time is predicted based on the second optical flow. The value of the pixel at the new position is calculated using bilinear interpolation or bicubic interpolation algorithms, generating a processed image frame. This step considers the motion direction and velocity of the pixels, ensuring the processed image frame reflects the scene content at the intermediate time. Pixels in the processed image frame are then weighted and fused. Specifically, for each pixel in the processed image frame, its contribution to the final intermediate image frame is adjusted according to its weight. Pixels with high weight (e.g., regions with high motion consistency) contribute more to the intermediate image frame, while pixels with low weight (e.g., regions with unreliable depth information or motion blur contribute less. Based on the above weighted fusion steps, the final intermediate image frame is generated. The intermediate image frame is located between the original adjacent frames and reflects the scene content at the intermediate time.

[0084] Intermediate image frames can capture subtle changes in an object during continuous motion, generating richer image details. The movement of objects in the video is more realistic, and the details are more vividly rendered.

[0085] In the embodiments of this disclosure, in order to facilitate a better understanding of the entire video processing process, such as Figure 3 As shown, Figure 3 This is a video processing flowchart provided in an embodiment of the present disclosure. A camera (i.e., a terminal) captures the current scene, and simultaneously utilizes the camera and a ToF distance sensor to acquire the corresponding RGB video stream (i.e., the first video) and a single-channel depth information stream, denoted as I and D, respectively. Frame extraction is performed to acquire the RGB (i.e., image frame) and depth information (i.e., the first depth information) of frames t and t+1. To enhance (i.e., correct) the depth information, and to facilitate a better understanding of this correction process, such as... Figure 4 As shown, Figure 4 This is a flowchart illustrating a depth information correction process provided in an embodiment of the present disclosure. and Blur processing is performed to suppress a large number of high-frequency noise points in the ToF data, resulting in a blurred depth map (i.e., filtered depth information). and To improve processing speed, the following steps can be taken: Perform 8x bilinear downsampling on the depth information; apply Gaussian blur to the downsampled image; perform 8x bilinear upsampling on the blurred result; and then... and Perform edge restoration. Because of this... and Blurring leads to inaccurate edge information, so it is necessary to use the original resolution information for edge restoration to obtain the restored result (i.e., second depth information). and The specific steps are as follows: Extract and Edge information; for each pixel within an 8x8 window at the edge, calculate... and The difference. If the difference exceeds the threshold. Then the value at that point is... The value is 0, otherwise it is 0. ; and The handling method and and The processing method is the same.

[0086] In the embodiments of this disclosure, a neural network is used to calculate optical flow and perform frame interpolation. To facilitate a better understanding of the entire optical flow processing process, such as... Figure 5 As shown, Figure 5 This is a flowchart of an optical flow processing method provided in an embodiment of the present disclosure. First, it uses... and Calculate the optical flow to obtain a coarse estimate of the optical flow (i.e., the first optical flow). and ; RGB image and a rough estimate of optical flow and Input into a neural network to calculate refined optical flow and And the occlusion mask information (i.e., weights). Insert intermediate frames (i.e., intermediate image frames). ,have:

[0087] Here, warp(•) represents the warp sampling operation. To avoid image quality anomalies such as holes, the calculated optical flow is the backward optical flow, representing the positional relationship of pixels in frame t+0.5 within the sampled frame.

[0088] To facilitate a better understanding of the structure of optical flow computing networks (i.e., neural networks), such as Figure 6 As shown, Figure 6 This is a schematic diagram of a neural network structure provided in an embodiment of the present disclosure. During network training, the optical flow outputs of the two sub-networks are trained using the same loss function, i.e., the perceptual difference between the interpolated intermediate frames and the real intermediate frames. The specific neural network weights are as follows:

[0089] in, For neural network weights, The mean absolute error, This is a structural similarity index.

[0090] Corresponding to the video processing method described above, the present invention also proposes a video processing apparatus. Since the apparatus embodiments of the present invention correspond to the method embodiments described above, details not disclosed in the apparatus embodiments can be referred to in the method embodiments described above, and will not be repeated here.

[0091] Figure 7 This is a schematic diagram of the structure of a video processing apparatus 300 provided in an embodiment of the present disclosure. The video processing apparatus includes: The acquisition unit 31 is used to acquire image frames in the first video captured by the terminal, and to acquire first depth information corresponding to the image frames collected by the terminal. The first processing unit 32 is used to perform correction processing on the first depth information to obtain the second depth information; The second processing unit 33 is used to perform frame interpolation processing on the first video according to the image frame and the second depth information to obtain the second video.

[0092] In summary, in the embodiments of this disclosure, the video processing apparatus proposed in this disclosure includes acquiring first depth information corresponding to image frames collected by a terminal, correcting the first depth information to obtain second depth information, thereby improving the accuracy of the second depth information. The second depth information can more realistically reflect the depth characteristics of the video scene. The first video is interpolated by combining the second depth information and the image frames to obtain the second video, thereby reducing the problem of interpolation errors or blurring caused by inaccurate depth information, and thus improving the effect of video processing.

[0093] Furthermore, in one possible implementation of the embodiments of this disclosure, such as Figure 8 As shown, the first processing unit 32 includes: The first processing module 321 is used to perform fuzzy filtering on the first depth information to obtain filtered depth information. The adjustment module 322 is used to perform edge adjustment on the filtered depth information to obtain the second depth information.

[0094] Furthermore, in one possible implementation of the embodiments of this disclosure, such as Figure 8 As shown, the adjustment module 322 is further used for: Extract the first edge information from the filtered depth information, and extract the second edge information from the first depth information; Based on the difference between the first edge information and the second edge information, edge adjustment is performed on the filtered depth information to obtain the second depth information.

[0095] Furthermore, in one possible implementation of the embodiments of this disclosure, such as Figure 8 As shown, the adjustment module 322 is further used for: If the difference is greater than a preset threshold, the first edge information in the filtered depth information is updated to the second edge information to obtain the second depth information; If the difference is less than or equal to the preset threshold, the filtered depth information is determined as the second depth information.

[0096] Furthermore, in one possible implementation of the embodiments of this disclosure, such as Figure 8 As shown, the second processing unit 33 includes: The generation module 331 is used to generate the first optical flow of the adjacent frames based on the adjacent depth information of the adjacent frames in the second depth information in the image frames of the first video; The generation module 331 is used to generate a second optical flow of the adjacent frame and a weight corresponding to the second optical flow based on the first optical flow and the adjacent frame; The second processing module 332 is used to perform frame interpolation processing on the first video according to the second optical flow and the weight to obtain the second video.

[0097] Furthermore, in one possible implementation of the embodiments of this disclosure, such as Figure 8 As shown, the second processing module 332 is further configured to: Based on the second optical flow and the weight, an intermediate image frame between the adjacent frames is generated; Based on the intermediate image frames, the first video is subjected to the frame interpolation process to obtain the second video.

[0098] Furthermore, in one possible implementation of the embodiments of this disclosure, such as Figure 8 As shown, the second processing module 332 is further configured to: The second optical flow and the adjacent frames are subjected to distorted sampling processing to obtain the processed image frame; The intermediate image frame is generated based on the processed image frame and the weight.

[0099] Since the apparatus provided in this embodiment corresponds to the methods provided in the above embodiments, the implementation of the methods is also applicable to the apparatus provided in this embodiment, and will not be described in detail in this embodiment.

[0100] The methods and apparatus provided in the embodiments of this application have been described above. To implement the functions of the methods provided in the embodiments of this application, the electronic device may include a hardware structure and software modules, and may implement the above functions in the form of a hardware structure, software modules, or a hardware structure plus software modules. One of the above functions may be executed in the form of a hardware structure, software modules, or a hardware structure plus software modules.

[0101] Figure 9 This is a block diagram illustrating an electronic device 400 for implementing the above-described video processing method according to an exemplary embodiment. For example, the electronic device 400 may be applied to servers, cloud environments, operational service platforms, various computer platforms, terminal systems, and web page systems.

[0102] Reference Figure 9 The electronic device 400 may include one or more of the following components: processing component 402, memory 404, power supply component 406, multimedia component 408, audio component 410, input / output (I / O) interface 412, sensor component 414, and communication component 416.

[0103] Processing component 402 typically controls the overall operation of electronic device 400, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 402 may include one or more processors 420 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 402 may include one or more modules to facilitate interaction between processing component 402 and other components. For example, processing component 402 may include a multimedia module to facilitate interaction between multimedia component 408 and processing component 402.

[0104] Memory 404 is configured to store various types of data to support the operation of electronic device 400. Examples of such data include instructions for any application or method operating on electronic device 400, contact data, phonebook data, messages, pictures, videos, etc. Memory 404 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0105] Power supply component 406 provides power to various components of electronic device 400. Power supply component 406 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 400.

[0106] Multimedia component 408 includes a screen that provides an output interface between electronic device 400 and a user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 408 includes a front-facing camera and / or a rear-facing camera. When electronic device 400 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0107] Audio component 410 is configured to output and / or input audio signals. For example, audio component 410 includes a microphone (MIC) configured to receive external audio signals when electronic device 400 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 404 or transmitted via communication component 416. In some embodiments, audio component 410 also includes a speaker for outputting audio signals.

[0108] I / O interface 412 provides an interface between processing component 402 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0109] Sensor assembly 414 includes one or more sensors for providing state assessments of various aspects of electronic device 400. For example, sensor assembly 414 may detect the on / off state of electronic device 400, the relative positioning of components such as the display and keypad of electronic device 400, changes in position of electronic device 400 or a component of electronic device 400, the presence or absence of user contact with electronic device 400, orientation or acceleration / deceleration of electronic device 400, and temperature changes of electronic device 400. Sensor assembly 414 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 414 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 414 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0110] Communication component 416 is configured to facilitate wired or wireless communication between electronic device 400 and other devices. Electronic device 400 can access wireless networks based on communication standards, such as WiFi, 2G or 3G, 4G LTE, 4G NR (NewRadio), or combinations thereof. In one exemplary embodiment, communication component 416 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 416 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0111] In an exemplary embodiment, the electronic device 400 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0112] Embodiments of this disclosure also provide a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to perform the methods described in the above embodiments of this disclosure.

[0113] Embodiments of this disclosure also provide a computer program product comprising a computer program executable by a programmable device, the computer program having, when executed by the programmable device, the method described in the foregoing embodiments of this disclosure.

[0114] For cases where electronic devices can be chips or chip systems, see [link to relevant documentation]. Figure 10 The diagram shows the structure of the chip. Figure 10 The chip shown includes a processor 501 and an interface 502. There can be one or more processors 501, and multiple interfaces 502.

[0115] Optionally, the chip also includes a memory 503 for storing necessary computer programs and data.

[0116] Those skilled in the art will also understand that the various illustrative logical blocks and steps listed in the embodiments of this application can be implemented by electronic hardware, computer software, or a combination of both. Whether such functionality is implemented through hardware or software depends on the specific application and the overall system design requirements. Those skilled in the art can implement the functionality using various methods for each specific application, but such implementation should not be construed as exceeding the scope of protection of the embodiments of this application.

[0117] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0118] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0119] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0120] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processing module, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (control method), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic device, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0121] It should be understood that various parts of the embodiments of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0122] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware, and the program can be stored in a computer-readable storage medium. When executed, the program includes one or a combination of the steps of the method embodiments.

[0123] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc.

[0124] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A video processing method, characterized in that, The method includes: Acquire image frames from the first video captured by the terminal, and acquire the first depth information corresponding to the image frames collected by the terminal; The first depth information is corrected to obtain the second depth information; Based on the image frames and the second depth information, the first video is subjected to frame interpolation to obtain the second video.

2. The method according to claim 1, characterized in that, The step of correcting the first depth information to obtain the second depth information includes: The first depth information is subjected to fuzzy filtering to obtain filtered depth information. The filtered depth information is adjusted at the edges to obtain the second depth information.

3. The method according to claim 2, characterized in that, The step of edge adjustment of the filtered depth information to obtain the second depth information includes: Extract the first edge information from the filtered depth information, and extract the second edge information from the first depth information; Based on the difference between the first edge information and the second edge information, edge adjustment is performed on the filtered depth information to obtain the second depth information.

4. The method according to claim 3, characterized in that, The step of adjusting the filtered depth information based on the difference between the first edge information and the second edge information to obtain the second depth information includes: If the difference is greater than a preset threshold, the first edge information in the filtered depth information is updated to the second edge information to obtain the second depth information; If the difference is less than or equal to the preset threshold, the filtered depth information is determined as the second depth information.

5. The method according to claim 1, characterized in that, The step of performing frame interpolation on the first video based on the image frame and the second depth information to obtain the second video includes: Based on the adjacent depth information of adjacent frames in the image frames of the first video in the second depth information, the first optical flow of the adjacent frames is generated; Based on the first optical flow and the adjacent frames, generate the second optical flow of the adjacent frames and the weights corresponding to the second optical flow; Based on the second optical flow and the weights, the first video is subjected to frame interpolation to obtain the second video.

6. The method according to claim 5, characterized in that, The step of performing frame interpolation on the first video based on the second optical flow and the weights to obtain the second video includes: Based on the second optical flow and the weight, an intermediate image frame between the adjacent frames is generated; Based on the intermediate image frames, the first video is subjected to the frame interpolation process to obtain the second video.

7. The method according to claim 6, characterized in that, The step of generating intermediate image frames between adjacent frames based on the second optical flow and the weights includes: The second optical flow and the adjacent frames are subjected to distorted sampling processing to obtain the processed image frame; The intermediate image frame is generated based on the processed image frame and the weight.

8. A video processing apparatus, characterized in that, The device includes: The acquisition unit is used to acquire image frames from a first video captured by the terminal, and to acquire first depth information corresponding to the image frames collected by the terminal. The first processing unit is used to correct the first depth information to obtain the second depth information; The second processing unit is used to perform frame interpolation processing on the first video based on the image frame and the second depth information to obtain the second video.

9. The apparatus according to claim 8, characterized in that, The first processing unit includes: The first processing module is used to perform fuzzy filtering on the first depth information to obtain filtered depth information. An adjustment module is used to perform edge adjustment on the filtered depth information to obtain the second depth information.

10. The apparatus according to claim 9, characterized in that, The adjustment module is also used for: Extract the first edge information from the filtered depth information, and extract the second edge information from the first depth information; Based on the difference between the first edge information and the second edge information, edge adjustment is performed on the filtered depth information to obtain the second depth information.

11. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.

12. A computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-7.

13. A chip, characterized in that, The chip includes a processing circuit and an interface circuit; wherein the interface circuit is used to read instructions and send the instructions to the processing circuit so that the processing circuit executes the method as described in any one of claims 1-7.

Images