Video processing method and apparatus thereof

Abstract

This application discloses a video processing method and apparatus, belonging to the field of video processing technology. The method includes acquiring a target frame of a target video and depth information of the target frame; performing motion estimation on the target frame based on historical frames in the target video relative to the target frame to determine a first motion vector of the target frame, wherein the resolution of the motion estimation is smaller than that of the target frame; updating the first motion vector based on the depth information of the target frame to obtain a second motion vector of the target frame; and registering the target frame with the historical frames based on the second motion vector.

Description

Technical Field

[0001] This application belongs to the field of video processing technology, specifically relating to a video processing method and apparatus. Background Technology

[0002] Since video requires at least 25 frames per second, and the exposure time of each frame cannot exceed 40 milliseconds, and the exposure time is directly related to the image signal-to-noise ratio, image noise is the main problem affecting video image quality.

[0003] Video denoising typically utilizes temporal information between frames, requiring the processing of multiple frames together. During video recording, movement by the photographer or hand-held camera shake causes spatial shifts between frames. Therefore, spatial registration of each frame is necessary before denoising. Registration algorithms commonly employ optical flow and block matching. These methods usually divide the image into multiple blocks; finer block divisions result in higher registration resolution, but also place higher demands on device performance, limiting their applicability. Summary of the Invention

[0004] The purpose of this application is to provide a video processing method and apparatus that can reduce the resolution of image registration and solve the problem of high device performance requirements for algorithms.

[0005] In a first aspect, embodiments of this application provide a video processing method, the method comprising:

[0006] Obtain the target frame of the target video, and the depth information of the target frame;

[0007] Motion estimation of the target frame is performed on the target frame based on historical frames in the target video relative to the target frame to determine the first motion vector of the target frame, wherein the resolution of the motion estimation is smaller than that of the target frame;

[0008] The first motion vector is updated based on the depth information of the target frame to obtain the second motion vector of the target frame;

[0009] The target frame is registered with the historical frame based on the second motion vector.

[0010] Secondly, embodiments of this application provide a video processing apparatus, the apparatus comprising:

[0011] The first acquisition module is used to acquire the target frame of the target video and the depth information of the target frame;

[0012] The first determining module is used to perform motion estimation on the target frame based on historical frames in the target video relative to the target frame, and determine a first motion vector of the target frame, wherein the resolution of the motion estimation is smaller than that of the target frame;

[0013] The first update module is used to update the first motion vector based on the depth information of the target frame to obtain the second motion vector of the target frame;

[0014] The first registration module is used to register the target frame with the historical frame based on the second motion vector.

[0015] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, wherein the memory stores programs or instructions that can run on the processor, and the programs or instructions, when executed by the processor, implement the video processing method as described in the first aspect.

[0016] Fourthly, embodiments of this application provide a readable storage medium storing a program or instructions that, when executed by a processor, implement the video processing method as described in the first aspect.

[0017] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the video processing method as described in the first aspect.

[0018] In a sixth aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the video processing method as described in the first aspect.

[0019] In this embodiment, video registration does not require fine segmentation. Motion estimation is performed at low resolution to determine the motion vector of the target frame. Then, depth information is used to assist in registration. This reduces the computational load of the motion estimation algorithm while ensuring registration accuracy, thereby reducing the algorithm's complexity, reducing the performance requirements of image registration, expanding the application scenarios of registration, and improving the video quality. Attached Figure Description

[0020] Figure 1 This is a flowchart of the video processing method provided in the embodiments of this application;

[0021] Figure 2 This is a schematic diagram of an image in the video processing method provided in the embodiments of this application;

[0022] Figure 3 This is a schematic diagram illustrating the determination of the first motion vector in the video processing method provided in this application embodiment;

[0023] Figure 4 This is a schematic diagram illustrating an application scenario of the video processing method provided in the embodiments of this application;

[0024] Figure 5 This is a schematic diagram of the structure of the video processing apparatus provided in the embodiments of this application;

[0025] Figure 6 This is one of the structural schematic diagrams of the electronic device provided in the embodiments of this application;

[0026] Figure 7 This is the second schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0027] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0028] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0029] The video processing method and apparatus provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0030] This application first provides a video processing method that can be applied to electronic devices such as mobile phones, tablets, laptops, wearable electronic devices (such as smartwatches), augmented reality (AR) / virtual reality (VR) devices, and in-vehicle devices. This application does not impose any limitations on this method.

[0031] Due to factors such as hand-held sway and object movement, multi-frame noise reduction is commonly used in video denoising. This not only reduces noise but also minimizes video blur. Multi-frame noise reduction involves capturing multiple images, aligning them, and then using the average value at each point for noise reduction. Therefore, the accuracy of the alignment has a significant impact on video quality.

[0032] Figure 1 This is a flowchart of the video processing method provided in this embodiment. For example... Figure 1 As shown, the video processing method includes the following:

[0033] Step 10: Obtain the target frame of the target video and the depth information of the target frame.

[0034] Images at any given moment can be acquired using electronic devices that include image sensors. Multiple images captured consecutively over a period of time can form a video. The target frame can be any image in the target video, and this image can be of various formats, such as RGB, RAW, or YUV images, etc.

[0035] Electronic devices can also determine depth information between themselves and objects through a depth sensing module. For example, a mobile phone's laser autofocus module can determine the distance between the object being photographed and the phone. Exemplarily, this depth information may include a high-precision depth map, such as depth information measured by a binocular vision module, which includes the relative distance between each object in the scene and the electronic device. The depth information may also include simple distance information, such as the distance between the target object and the electronic device measured by a laser ranging module. Alternatively, the depth information may also include information indicating distance, such as "far" or "near," which is not specifically limited in this embodiment.

[0036] Step 20: Perform motion estimation on the target frame based on historical frames in the target video relative to the target frame, and determine the first motion vector of the target frame. The resolution of the motion estimation is smaller than that of the target frame.

[0037] In this embodiment, the historical frame is an image in the target video that occurs before the target frame, and the target frame and the historical frame can be adjacent images. For example, motion estimation can include block matching algorithms based on the Sum of Absolute Difference (SAD), grayscale projection algorithms, etc., and this embodiment does not impose any special limitations on these. Motion estimation requires dividing the historical frame and the target frame into multiple blocks according to a certain resolution, and determining the local motion vector of each block, i.e., the first motion vector. In this embodiment, the resolution used for motion estimation can be much lower than the resolution of the target frame. For example, if the resolution of the target frame is 1080×1920, the resolution of motion estimation can be low resolutions such as 4×4 or 4×8. The higher the resolution of motion estimation, the more refined the obtained motion vector; the lower the resolution, the coarser the corresponding motion vector. For example, when the resolution of the target frame is 1080×1920 and the resolution of motion estimation is 4×4, a 4×4 motion vector of the target frame can be obtained, which is much lower than the resolution of the target frame. Therefore, the scale of the blocks with the same motion vector in the target frame is larger.

[0038] For example, the first motion vector of the target frame is determined using grayscale projection. First, the two H×W images, the historical frame and the target frame, are divided into m×n blocks of equal size. 1≤m≤H, 1≤n≤W. H and W are the height and width of the target frame and the historical frame, respectively. The first motion vector Vx of the image blocks at corresponding positions in the target frame and the historical frame is then calculated in the horizontal and vertical directions using grayscale projection. i,j Vy i,j x i,j Let (i, j) represent the horizontal motion vector of the image patch at (i, j), where 1 ≤ i ≤ m and 1 ≤ j ≤ n.

[0039] Taking m=n=4 as an example, such as Figure 2 As shown, image 21 is a historical frame, and image 22 is the target frame. Images 21 and 22 are each divided into 4×4 image blocks. The grayscale projection method is used to calculate the spatial offset vectors in the x and y directions for corresponding positions of image blocks in images 21 and 22, such as image block 211 and image block 221. First, the sum of pixel values ​​in the x and y directions of image block 211 is calculated using formula (1):

[0040]

[0041] Among them, B t0 Let (x, y) be the pixel value at point (x, y) in image patch 211. This is the sum of pixel values ​​along the x-direction. This is the sum of pixel values ​​along the y-direction. Where M = H / 4 and N = W / 4 are the height and width of image block 211. Applying the same processing to image block 221 using the above formula (1) yields the corresponding... Then, the SAD of image block 211 and image block 221 under different offsets in the x and y directions is calculated respectively. The specific calculation formula is as follows:

[0042]

[0043] Here, the value of k is pre-set to be from 1 to L, where L∈(2, N-1), but can also be set to other ranges depending on the actual situation. δ is the k corresponding to the minimum SAD value, and N is the width of the image patch. SAD at different offsets k can be calculated using this method. Figure 3As shown, curve S1 is the sum of pixel values ​​in the x-direction of the image block in the historical frame, and curve S2 is the sum of pixel values ​​in the x-direction of the corresponding image block in the target frame. The absolute difference (SAD) between curves S1 and S2 at k=1 is point 30. By calculating the SAD for different values ​​of k, curve S3 can be obtained. The value of k at which SAD is minimized is determined; this value is the offset in the x-direction between the target frame and the historical frame. Then, the actual motion vector Vx of the target frame relative to the historical frame is:

[0044]

[0045] (L+1) / 2 is the offset center of the target frame. That is, the starting point of the target frame offset is (L+1) / 2. According to the above formula (2), it can be determined that when the target frame offset is δ, SAD is the smallest, and the actual first motion vector of the target frame relative to the historical frame is Vx. For example, when L is moved to the right in the range of [0,9], the offset center is 5. If SAD is the smallest when the target frame is offset by 3, then the first motion vector of the target frame is 5-3=2, that is, it moves 2 pixels to the right. The same applies to the Y direction.

[0046] Repeating the above process for each block yields a first motion vector with a resolution of 4×4, as follows:

[0047]

[0048] Step 30: Update the first motion vector based on the depth information of the target frame to obtain the second motion vector of the target frame.

[0049] The distance between an object and the camera is related to the distance it moves in the image. When acquiring images, there are generally two scenarios: one where the foreground is large and the object distance is large, and the other where the foreground is small and the object distance is small. For example, an object occupying more than half of the image frame is considered a large foreground, while one occupying less than half is considered a small foreground. Figure 4 As shown, in Figure 4 In the scene shown in (a), the car, trees, and houses are far from the camera, i.e., the object distance is large; Figure 4 In the scene shown in (b), the car is close to the camera, meaning the object distance is small. At a large object distance, the foreground's motion relative to the camera is less than at a small object distance. In other words, for the same amount of foreground motion, the smaller the object distance, the more pixels the object moves in the image. The more pixels move, the stronger the drag effect between the foreground and background. Furthermore, a smaller object distance results in a shallower depth of field and a more blurred background (blurred background). A blurred background reduces sensitivity to registration errors.

[0050] Therefore, based on the depth information of the image, the weights for the foreground and background can be determined. Then, the corresponding parts of the first motion vector are adjusted according to these weights, thereby improving the overall image registration accuracy. For example, for... Figure 4 In the case shown in (b), the car is the foreground. When the foreground moves relative to the camera, the first motion vector corresponding to the background is incorporated into the foreground according to a certain weight, thereby improving the overall clarity of the image.

[0051] Specifically, first determine the foreground and background vectors in the first motion vector. Using the least squares method, the first motion vector can be divided into a foreground vector and a background vector, calculated as follows:

[0052]

[0053] Formula (4) above can determine ε when the minimum value is obtained. After obtaining the result of the least squares method through formula (4) above, the value greater than V can be used. thre The first motion vector is defined as the foreground vector, denoted as . Less than V thre The first motion vector is defined as the background vector, denoted as . The same applies to the Y direction.

[0054] In addition, foreground and background vectors can also be segmented using other methods, such as support vector machines, k-means clustering, etc., but this implementation is not limited to these.

[0055] Next, weighting coefficients are determined based on the depth information of the target frame. These coefficients are then used to weight the foreground and background vectors separately to obtain the second motion vector. For example, if the depth information of the target frame is a depth map, the depth information of each region can be determined based on this map. A depth map uses the pixel value of each point to represent the distance of an object; for example, the farther the object is from the camera, the closer its color value is to 255, and the closer it is, the closer it is to 0. The formula for determining the depth information of each image patch in the target frame is as follows:

[0056]

[0057] Among them, D i,j Image block B representing the target frame i,j The depth information, MED(·), is the median function used to determine image patch B. i,j The median of the depth map corresponding to the image patch is used as the depth information of that image patch. The depth information of each image patch can be calculated using formula (5).

[0058] For example, after determining the depth information, this depth information can be converted into weights between 0 and 1, i.e., weighting coefficients, and then the foreground and background vectors are weighted. Since the foreground and background have different imaging sharpness, when the foreground is close, its movement will create a large distance in the image, resulting in noticeable drag between it and the background. Therefore, for small object distances, the corresponding background vector can be incorporated into the foreground vector according to the weighting coefficients, thereby reducing the drag between the foreground and background and improving the image registration accuracy. For large object distances, the foreground vector can also be incorporated into the background vector according to the weighting ratio. Registration by combining foreground and background can compensate for the insufficient accuracy of motion estimation algorithms and improve the registration accuracy.

[0059] Specifically, when determining the weighting coefficients, the target depth range is determined based on manual configuration. This target depth range includes the maximum depth d that is preset manually. max and minimum depth d min The target depth range can be [0,9], [5,10], etc., and this embodiment does not impose any special limitation on it. The depth information mentioned above is converted into depth values ​​within the target depth range through the target depth range, and then the corresponding weighting coefficient is calculated based on the depth value. For example, the weighting coefficient of each image block of the target frame is determined by the following formula (6):

[0060]

[0061] Here, clip(·) is a constraint function used to apply depth information D. i,j The depth is limited to between the maximum and minimum depths. Formula (6) above converts the depth values ​​between the maximum and minimum depths into a weighted coefficient α between 0 and 1. i,j The larger the depth value, the larger the weighting coefficient.

[0062] For example, if the depth information includes the distance to the foreground, the depth information can be converted to 0 or 1 as a weighting coefficient. For instance, if an electronic device acquires the depth information of a target frame based on a laser autofocus module, the laser autofocus module can provide information about the distance between the foreground and the camera. As shown in the figure, when the camera acquires an image, the laser autofocus module 401 can detect the distance between the person (foreground) and the camera. When the electronic device obtains this distance, if the distance is greater than a preset value, the corresponding weighting coefficient is 1; otherwise, the weighting coefficient is 0.

[0063] The foreground and background vectors are weighted separately according to the weighting coefficient to obtain the second motion vector. If the first motion vector is a foreground vector, then to reduce drag between the foreground and background, the background vector can be merged with the foreground vector according to the weighting ratio to obtain a weighted foreground vector, i.e., the second motion vector. Specifically, the average value of all foreground vectors in the target frame is determined, denoted as the first average value, and the second average value of the background vectors is determined. During the weighting process, the foreground vector can be weighted with the second average value of the background vectors, while the background vector is weighted with the first average value of the foreground vectors.

[0064] For example, if the first motion vector is the foreground vector, then the weighting is performed according to the following formula (7):

[0065]

[0066] in, This represents the second motion vector after horizontal weighting of the image patch. This represents the first horizontal motion vector of the image patch before weighted processing. This is the second average value in the horizontal direction. The processing method for the vertical direction is the same. If the depth of the foreground is greater, such as... Figure 4 In the scenario shown in (a), the larger the weighting coefficient, the smaller the difference between the foreground and background offset vectors, and the more the weighting is biased towards the background vector. If the foreground depth is small, such as... Figure 4 In the scenario shown in (b), the smaller the weighting coefficient, the better. When the foreground is close, the camera focus is on the foreground, and the background will be blurred. This reduces the sensitivity to registration errors, so the weighting is more biased towards the foreground vector itself.

[0067] If the first motion vector is the background vector, then weighting can be performed according to the following formula (8):

[0068]

[0069] in, This is the average value of the foreground vectors, i.e., the first average value. When the background depth is greater, the weighting coefficient is larger, and the weighting is biased towards the background vector itself. When the background depth is less, the difference between the background and the foreground is smaller, the weighting coefficient is smaller, and the weighting is biased towards the foreground vector.

[0070] For example, when the weighting coefficient is α i,jWhen the value is 1, the object has a large depth of field. The weighted foreground vector has the same second average value as the background vector. The background vector is unaffected by the weighting process, meaning the second motion vector remains consistent with the first motion vector. This implementation cleverly utilizes the foreground-background offset characteristic to adjust the motion vector determined by the motion estimation algorithm, reducing the image "drag" problem caused by low motion estimation resolution. It transfers unavoidable configuration errors to visually insensitive areas, improving the visual effect of the image. Furthermore, the weighting process does not add extra equipment, the algorithm is relatively simple, and it does not burden the performance of electronic devices, thus expanding the application range of registration.

[0071] Next, continue to refer to Figure 1 Step 40: Register the target frame with the historical frames based on the second motion vector.

[0072] The offset vector of each pixel in the target frame can be determined based on the second motion vector, thereby moving the target frame so that it is in the same spatial domain viewpoint as the historical frames, facilitating noise reduction of both the historical and target frames. Video denoising algorithms can be used to denoise the registered target video. These algorithms can include multi-frame denoising algorithms, neural network-based denoising algorithms, and various other algorithms; this implementation is not limited to these.

[0073] For example, the second motion vector can be amplified based on an interpolation algorithm so that its resolution is the same as that of the target frame. Interpolation algorithms can include bilinear interpolation, nearest neighbor interpolation, etc., and this implementation is not limited to these. The interpolated second motion vector has the same resolution as the target frame, and each point in the second motion vector corresponds to each point in the target frame. Based on the second motion vector, the offset vector of each pixel in the target frame can be determined, i.e., the distance and direction to be moved. Then, the pixels in the target frame are moved according to this offset vector to obtain the registered target frame. The registered target frame is an image with the same spatial domain viewpoint as the historical frame.

[0074] Furthermore, the video processing method provided in this application embodiment can be executed by a video processing device. Below, the video processing device provided in this application embodiment will be described using the execution of the video processing method within a video processing device as an example.

[0075] like Figure 5As shown, the video processing apparatus 50 provided in this embodiment may include a first acquisition module 51, a first determination module 52, a first update module 53, and a first registration module 54. Specifically, the first acquisition module 51 is used to acquire a target frame of a target video and the depth information of the target frame; the first determination module 52 is used to perform motion estimation on the target frame based on historical frames in the target video relative to the target frame, and determine a first motion vector of the target frame, wherein the resolution of the motion estimation is smaller than that of the target frame; the first update module 53 is used to update the first motion vector based on the depth information of the target frame to obtain a second motion vector of the target frame; and the first registration module 54 is used to register the target frame with the historical frames based on the second motion vector.

[0076] The video processing apparatus provided in this embodiment utilizes motion vectors determined by a low-resolution motion estimation algorithm, which reduces the accuracy requirements of the motion estimation algorithm and thus lowers the performance requirements of the registration equipment, thereby expanding its application range. Furthermore, by taking advantage of the characteristics of the foreground and background moving relative to the camera, the motion vectors obtained from motion estimation are adjusted, thereby improving the registration accuracy.

[0077] In an exemplary embodiment, the first update module 53 includes: a first segmentation module, used to determine the foreground vector and the background vector in the first motion vector; a second determination module, used to determine weighting coefficients based on the depth information of the target frame; and a first weighting module, used to perform weighting processing on the foreground vector and the background vector respectively based on the weighting coefficients to obtain the second motion vector.

[0078] In an exemplary embodiment, the second determining module includes: a first conversion module, used to obtain a preset target depth range and convert the depth information into depth values ​​within the target depth range; and a third determining module, used to determine a weighting coefficient based on the depth values ​​within the target depth range, wherein the weighting coefficient ranges from 0 to 1, and the larger the depth value, the larger the corresponding weighting coefficient value.

[0079] In an exemplary embodiment, the first weighting module includes: a fourth determining module, used to determine a first average value of the foreground vector and a second average value of the background vector; a second weighting module, used to weight the foreground vector and the second average value according to weighting coefficients to obtain a second motion vector corresponding to the foreground vector; and a third weighting module, used to weight the background vector and the first average value based on weighting coefficients to obtain a second motion vector corresponding to the background vector.

[0080] In an exemplary embodiment, the first registration module 54 includes: a fifth determining module, configured to determine the offset vector of each pixel in the target frame based on the second motion vector; and a first moving module, configured to move the pixel based on the offset vector to obtain the registered target frame.

[0081] In an exemplary embodiment, the video processing apparatus further includes: a first noise reduction module, used to reduce noise in the target video based on historical frames and the target frame.

[0082] The video processing device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television set (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the device.

[0083] The video processing device in this application embodiment can be a device with an operating system. The operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system.

[0084] The video processing device provided in this application embodiment can achieve... Figure 1 The various processes implemented in the method embodiments are not described in detail here to avoid repetition.

[0085] Optionally, such as Figure 6 As shown in the figure, this application embodiment also provides an electronic device 600, including a processor 601 and a memory 602. The memory 602 stores a program or instructions that can run on the processor 601. When the program or instructions are executed by the processor 601, they implement the various steps of the above-described video processing method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0086] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0087] Figure 7 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.

[0088] The electronic device 700 includes, but is not limited to, components such as: radio frequency unit 701, network module 7102, audio output unit 703, input unit 704, sensor 705, display unit 706, user input unit 707, interface unit 708, memory 709, and processor 710.

[0089] Those skilled in the art will understand that the electronic device 700 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 710 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 7 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0090] The processor 710 is used to execute the video processing method described above. For example, the processor 710 can be used to acquire a target frame of a target video and the depth information of the target frame; perform motion estimation on the target frame based on historical frames in the target video relative to the target frame to determine a first motion vector of the target frame, wherein the resolution of the motion estimation is smaller than that of the target frame; update the first motion vector based on the depth information of the target frame to obtain a second motion vector of the target frame; and register the target frame with the historical frames based on the second motion vector.

[0091] Input unit 704 may include a graphics processing unit (GPU) 7041 and a microphone 7042. The GPU 7041 processes image data of still images or videos acquired by an image capture device (such as a camera) in video capture mode or image capture mode. Display unit 706 may include a display panel 7061, which may be configured as a liquid crystal display, organic light-emitting diode, or the like. User input unit 707 includes at least one of a touch panel 7071 and other input devices 7072. Touch panel 7071 is also called a touch screen. Touch panel 7071 may include a touch detection device and a touch controller. Other input devices 7072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be described in detail here.

[0092] The memory 709 can be used to store software programs and various data. The memory 709 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 709 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 709 in the embodiments of this application includes, but is not limited to, these and any other suitable types of memory.

[0093] Processor 710 may include one or more processing units; optionally, processor 710 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 710.

[0094] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described video processing method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0095] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0096] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above video processing method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0097] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0098] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the video processing method embodiments described above, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0099] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0100] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0101] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A video processing method, characterized in that, include: Obtain the target frame of the target video, and the depth information of the target frame; Motion estimation of the target frame is performed on the target frame based on historical frames in the target video relative to the target frame to determine the first motion vector of the target frame, wherein the resolution of the motion estimation is smaller than that of the target frame; The first motion vector is updated based on the depth information of the target frame to obtain the second motion vector of the target frame; The target frame is registered with the historical frame based on the second motion vector; The step of updating the first motion vector based on the depth information of the target frame to obtain the second motion vector of the target frame includes: Determine the foreground vector and background vector in the first motion vector; The weighting coefficients are determined based on the depth information of the target frame; The foreground vector and the background vector are weighted according to the weighting coefficients to obtain the second motion vector.

2. The video processing method according to claim 1, characterized in that, Determining the weighting coefficients based on the depth information of the target frame includes: Based on the target depth range, the depth information is converted into a depth value within the target depth range; The weighting coefficient is determined based on the depth values ​​within the target depth range. The weighting coefficient ranges from 0 to 1, and the larger the depth value, the larger the corresponding weighting coefficient value.

3. The video processing method according to claim 1, characterized in that, The step of weighting the foreground vector and the background vector according to the weighting coefficients to obtain the second motion vector includes: Determine a first average value of the foreground vector and a second average value of the background vector; The foreground vector and the second average value are weighted according to the weighting coefficient to obtain the second motion vector corresponding to the foreground vector; The background vector and the first average value are weighted based on the weighting coefficients to obtain the second motion vector corresponding to the background vector.

4. The method according to claim 1, characterized in that, The registration of the target frame with the historical frame based on the second motion vector includes: The offset vector of each pixel in the target frame is determined based on the second motion vector; The pixel is moved based on the offset vector to obtain the registered target frame.

5. A video processing apparatus, characterized in that, include: The first acquisition module is used to acquire the target frame of the target video and the depth information of the target frame; The first determining module is used to perform motion estimation on the target frame based on historical frames in the target video relative to the target frame, and determine a first motion vector of the target frame, wherein the resolution of the motion estimation is smaller than that of the target frame; The first update module is used to update the first motion vector based on the depth information of the target frame to obtain the second motion vector of the target frame; The first registration module is used to register the target frame with the historical frame based on the second motion vector; The first update module includes: The first segmentation module is used to determine the foreground vector and background vector in the first motion vector; The second determining module is used to determine the weighting coefficients based on the depth information of the target frame; The first weighting module is used to weight the foreground vector and the background vector according to the weighting coefficients to obtain the second motion vector.

6. The video processing apparatus according to claim 5, characterized in that, The second determining module includes: The first conversion module is used to convert the depth information into a depth value within the target depth range according to the target depth range; The third determining module is used to determine the weighting coefficient based on the depth values ​​within the target depth range. The weighting coefficient ranges from 0 to 1, and the larger the depth value, the larger the corresponding weighting coefficient value.

7. The video processing apparatus according to claim 5, characterized in that, The first weighting module includes: The fourth determining module is used to determine the first average value of the foreground vector and the second average value of the background vector; The second weighting module is used to weight the foreground vector and the second average value according to the weighting coefficient to obtain the second motion vector corresponding to the foreground vector; The third weighting module is used to perform weighting processing on the background vector and the first average value based on the weighting coefficients to obtain the second motion vector corresponding to the background vector.

8. The video processing apparatus according to claim 5, characterized in that, The first registration module includes: The fifth determining module is used to determine the offset vector of each pixel in the target frame based on the second motion vector; The first moving module is used to move the pixel based on the offset vector to obtain the registered target frame.

Images