Video processing method and device, electronic equipment and readable storage medium
By interpolation of dual-camera videos, the frame rate and fluency of 3D videos are improved, and the problem of 3D videos in AR glasses is solved, the user experience is improved and the performance of electronic devices is optimized.
Patent Information
- Application Number
- CN202411115686.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-14
- Publication Date
- 2024-11-22
AI Technical Summary
When watching dual-camera videos with AR glasses, the 3D video image is not smooth and causes users to feel dizzy, and the existing technology is difficult to effectively solve this problem.
By acquiring the videos collected by the first camera and the second camera, and performing frame interpolation processing between adjacent frames, a high frame rate third and fourth videos are obtained, and a three-dimensional video is output based on these videos, thereby improving the 3D video. fluency.
The frame rate of dual-camera video is improved, ensuring the smoothness of 3D video displayed in AR devices, improving user viewing experience, and avoiding the power consumption and storage load problems caused by high frame rate videos by electronic devices.
Smart Images

Figure CN119011797A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of image technology, and specifically relates to a video processing method, device, electronic device and readable storage medium. Background Art
[0002] The technical solution for binocular three-dimensional (3D) video is to use a mobile phone to collect dual-camera video, and then watch it with augmented reality (AR) glasses to simulate the parallax of the left and right eyes through the dual-camera video to achieve a 3D effect.
[0003] In the prior art, when a user watches a dual-camera video on an AR glasses, the 3D video image may not be smooth, causing a certain sense of dizziness to the user. Summary of the invention
[0004] The purpose of the embodiments of the present application is to provide a video processing method that can solve the problem that when a user watches a dual-camera video on an AR glasses, the 3D video picture is not smooth, thereby causing a certain sense of dizziness to the user.
[0005] In a first aspect, an embodiment of the present application provides a video processing method, the method comprising: obtaining a first video captured by a first camera and a second video captured by a second camera; performing frame interpolation processing between two adjacent frames in the first video to obtain a third video, and performing frame interpolation processing between two adjacent frames in the second video to obtain a fourth video; and outputting a three-dimensional video based on the third video and the fourth video.
[0006] In a second aspect, an embodiment of the present application provides a video processing device, which includes: an acquisition module, used to acquire a first video captured by a first camera and a second video captured by a second camera; a processing module, used to perform frame interpolation processing between two adjacent frames in the first video to obtain a third video, and to perform frame interpolation processing between two adjacent frames in the second video to obtain a fourth video; and an output module, used to output a three-dimensional video based on the third video and the fourth video.
[0007] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps described in the first aspect are implemented.
[0008] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.
[0009] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.
[0010] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.
[0011] In an embodiment of the present application, in a shooting scene, the first camera captures an image to obtain a first video, and the second camera captures an image to obtain a second video. Further, the first video and the second video are obtained, and interpolation processing is performed between adjacent frames of the first video to obtain a third video, and interpolation processing is performed between adjacent frames of the second video to obtain a fourth video. Based on this, the third video is a high frame rate video compared to the first video, and the fourth video is a high frame rate video compared to the second video. Finally, according to the third video and the fourth video, a 3D video is output, so as to achieve the effect of displaying a dual-camera video with a higher frame rate. In particular, in an AR display scene, a 3D video can be displayed in an AR device to ensure the smoothness of the displayed 3D video and enhance the user's viewing experience. It can be seen that the embodiment of the present application can increase the frame rate of the dual-camera video, and in a 3D display scene, the amount of information received by the user's eyes can be increased, so as to better present a smooth 3D effect. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] Figure 1 is one of the flow charts of the video processing method of some embodiments of the present application;
[0013] Figure 2 This is the second flowchart of the video processing method of some embodiments of the present application;
[0014] Figure 3 is a third flowchart of the video processing method of some embodiments of the present application;
[0015] Figure 4 is one of the display schematic diagrams of the electronic device in some embodiments of the present application;
[0016] Figure 5 This is a second display schematic diagram of an electronic device in some embodiments of the present application;
[0017] Figure 6 is an operation diagram of a video processing method according to some embodiments of the present application;
[0018] Figure 7 is a block diagram of a video processing device according to some embodiments of the present application;
[0019] Figure 8 Schematic diagram of the hardware structure of the electronic device of some embodiments of the present application;
[0020] Fig. 9 It is a schematic diagram of the hardware structure of an electronic device of some embodiments of the present application. DETAILED DESCRIPTION
[0021] The following will be combined with the drawings of the embodiments of the present application to clearly describe the technical solutions of the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present application belong to the scope of protection of this application.
[0022] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally a class, and the number of objects is not limited. For example, the target object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated before and after are in an "or" relationship.
[0023] The video processing method provided in the embodiment of the present application is described in detail below through specific embodiments and their application scenarios in conjunction with the accompanying drawings.
[0024] like Figure 1 As shown, a flow chart of a video processing method according to an embodiment of the present application is shown. Taking the method applied to an electronic device as an example, the method includes: Step 110: obtaining a first video captured by a first camera and a second video captured by a second camera.
[0025] Usually, a video resolution of 1080p and a video frame rate of 30FPS can meet the shooting requirements. For 3D videos, the amount of information received by both eyes needs to be large enough to better present a smooth 3D effect. In order to improve the smoothness of 3D video images, it can be achieved from the perspective of increasing the frame rate of the video.
[0026] Optionally, the first camera is a main camera and the second camera is a secondary camera.
[0027] In addition, in order to avoid the problems of power consumption, storage, and performance load caused by the electronic device directly shooting a high frame rate video, the video shot by the electronic device is still a video with a normal frame rate, that is, the first video and the second video are videos with a normal frame rate. For the sake of distinction, the first video and the second video are both considered to be low frame rate videos.
[0028] For reference, the electronic device includes an external system on chip (SOC), so that the SOC can obtain the first video and the second video. Further, the SOC performs frame insertion processing on the first video and the second video respectively. In this way, on the basis of improving the video frame rate, it can ensure that the electronic device will not suffer problems such as power consumption, storage, and performance load.
[0029] Step 120: interpolate between two adjacent frames in the first video to obtain a third video, and interpolate between two adjacent frames in the second video to obtain a fourth video.
[0030] In this step, for the first video, interpolation processing is performed between every two adjacent frames; similarly, for the second video, interpolation processing is performed between every two adjacent frames.
[0031] Optionally, in the frame interpolation process, an interpolation algorithm is used, wherein the interpolation algorithm is an optical flow estimation method based on brightness difference.
[0032] Based on this, for the sake of easy distinction, relative to the first video and the second video, the third video and the fourth video are both considered to be high frame rate videos.
[0033] For example, if the frame rates of the first video and the second video are 30 FPS, the frame rates of the third video and the fourth video may be 60 FPS.
[0034] Step 130: Output a three-dimensional (3D) video based on the third video and the fourth video.
[0035] Optionally, based on a data connection between the electronic device and the display device, or the electronic device itself is a display device, the scenario of outputting a three-dimensional video may be playing the 3D video in the display device.
[0036] For example, the display device is an AR head display device.
[0037] It should be noted that 3D video overlaps the images of two videos to produce a 3D stereoscopic effect. When users wear AR head-mounted devices to watch, they feel as if they are in the scene. Among them, the two overlapping video images are not exactly the same, but are shot with two different cameras. The principle of 3D video is actually the same as that of the human eye. The reason why the world seen by the naked eye has a three-dimensional sense is that the slight angle difference between the two eyes is transmitted to the brain through the retina, so that the front and back distance of the scene can be distinguished, thereby producing a strong sense of three-dimensionality. When shooting 3D video, two cameras are used to imitate the human eye.
[0038] Optionally, a frame image of the third video at a certain moment is overlapped with a frame image of the fourth video at the same moment and displayed on a display device.
[0039] Correspondingly, the external SOC transmits the video data of the third video and the fourth video to the display module, so that the display module plays the 3D video according to the video data. The display module here can be an external display device or a display structure of the electronic device itself.
[0040] In an embodiment of the present application, in a shooting scene, the first camera captures an image to obtain a first video, and the second camera captures an image to obtain a second video. Further, the first video and the second video are obtained, and interpolation processing is performed between adjacent frames of the first video to obtain a third video, and interpolation processing is performed between adjacent frames of the second video to obtain a fourth video. Based on this, the third video is a high frame rate video compared to the first video, and the fourth video is a high frame rate video compared to the second video. Finally, according to the third video and the fourth video, a 3D video is output, so as to achieve the effect of displaying a dual-camera video with a higher frame rate. In particular, in an AR display scene, a 3D video can be displayed in an AR device to ensure the smoothness of the displayed 3D video and enhance the user's viewing experience. It can be seen that the embodiment of the present application can increase the frame rate of the dual-camera video, and in a 3D display scene, the amount of information received by the user's eyes can be increased, so as to better present a smooth 3D effect.
[0041] In the process of the video processing method of another embodiment of the present application, the frame rate of the first video is the same as the frame rate of the second video. For example, the frame rates of the first video and the second video are both 30 FPS.
[0042] Correspondingly, step 120 includes:
[0043] Sub-step A1: synthesize a first video frame according to the Nth frame image and the N+1th frame image of the first video, where N is a positive integer.
[0044] Step A1 is a repeated step, and the value of N is constantly changing.
[0045] like Figure 2 As shown, step 201 corresponds to step A1.
[0046] For example, first, N=1, the interpolation algorithm is used for the first frame image and the second frame image of the first video to synthesize a first video frame. Then, N=2, the interpolation algorithm is used for the second frame image and the third frame image of the first video to synthesize a first video frame. And so on.
[0047] It should be noted that, when interpolation is performed on the first video, the synthesized frame image is defined as the first video frame; similarly, in subsequent steps, interpolation is performed on the second video, and the synthesized frame image is defined as the second video frame. This embodiment takes the synthesis of one of the first video frames and one of the second video frames as an example, and does not specifically limit the number of the first video frames and the second video frames.
[0048] Sub-step A2: Obtaining a relative timestamp of the first video frame, wherein the relative timestamp of the first video frame is used to indicate that the first video frame is located between the Nth frame image and the N+1th frame image.
[0049] In this step, during the process of synthesizing a first target frame, that is, during the process of performing the frame interpolation algorithm based on the main shooting channel, a relative timestamp is recorded, and the relative timestamp is used to guide the frame interpolation algorithm based on the auxiliary shooting channel.
[0050] Sub-step A3: Determine the Mth frame image and the M+1th frame image of the second video according to the relative timestamp of the first video frame, where M is a positive integer.
[0051] like Figure 2 As shown, step 202 corresponds to step A3.
[0052] In this step, based on the relative timestamp of the first video frame, the position where the interpolation process needs to be performed can be found in the second video. For example, if the relative timestamp of a first video frame is used to indicate that the first video frame is located between the first frame image and the second frame image of the first video, the first frame image and the second frame image of the second video are found to perform the interpolation process between the first frame image and the second frame image of the second video.
[0053] Correspondingly, in this step, the relative timestamp of the first video frame is used to indicate that the first video frame is located between the Nth frame image and the N+1th frame image, and the relative timestamp is also used to indicate that the second video frame to be synthesized is located between the Mth frame image and the M+1th frame image.
[0054] Sub-step A4: synthesize a second video frame according to the Mth frame image and the M+1th frame image.
[0055] like Figure 2As shown, step 203 corresponds to step A4.
[0056] In this step, under the guidance of the relative timestamp of the first video frame, the interpolation position is determined in the second video, so that the interpolation process is performed at the interpolation position.
[0057] In this embodiment, in the interpolation process, optical flow estimation is performed under the guidance of relative timestamps, which can actively reduce the difference between the interpolated frames in the primary and secondary camera channels and improve the probability of interpolation consistency.
[0058] In the process of the video processing method of another embodiment of the present application, after synthesizing the second video frame, as shown in FIG. Figure 2 In step 204, a consistency check is performed on the first video frame and the second video frame associated with the same relative timestamp to ensure that the frame insertion is successful. Correspondingly, step 120 also includes:
[0059] Sub-step B1: Calculate the optical flow between the Nth frame image and the first video frame, and the optical flow between the first video frame and the N+1th frame image to obtain a first optical flow.
[0060] Specifically, in this embodiment, when a first video frame is synthesized between a first frame image and a second frame image of a first video, and a second video frame is synthesized between a first frame image and a second frame image of a second video, it is necessary to detect whether the first video frame and the second video frame are consistent.
[0061] like Figure 3 As shown, step 301 corresponds to step B1.
[0062] The N_t-th frame image is synthesized between the N-th frame image and the N+1-th frame image of the first video, and the optical flow from the N-th frame image to the N_t-th frame image and the optical flow from the N_t-th frame image to the N+1-th frame image are calculated, and these two parts of the optical flow are accumulated as a group of optical flows, namely, the first optical flow.
[0063] Sub-step B2: Calculate the optical flow between the Mth frame image and the second video frame, and the optical flow between the second video frame and the M+1th frame image to obtain a second optical flow.
[0064] like Figure 3 As shown, step 302 corresponds to step B2.
[0065] The M_t-th frame image is synthesized between the M-th frame image and the M+1-th frame image of the second video, and the optical flow from the M-th frame image to the M_t-th frame image and the optical flow from the M_t-th frame image to the M+1-th frame image are calculated, and these two parts of the optical flow are accumulated as a group of optical flows, namely, the second optical flow.
[0066] Sub-step B3: Calculate the degree of linear correlation based on the first optical flow and the second optical flow.
[0067] like Figure 3 As shown, step 303 corresponds to step B3.
[0068] In this step, the first optical flow is taken as a set of data, the second optical flow is taken as a set of data, and the linear correlation degree between the two sets of data is calculated.
[0069] Sub-step B4: When the degree of linear correlation is less than or equal to the degree threshold, a third video is obtained based on the Nth frame image, the first video frame, and the N+1th frame image, and a fourth video is obtained based on the Mth frame image, the second video frame, and the M+1th frame image.
[0070] In this step, if the linear correlation degree is less than or equal to the degree threshold, the two-way interpolation frames are consistent, and the primary and secondary camera interpolation frames are output.
[0071] For reference, when the linear correlation degree is less than or equal to the degree threshold, such as Figure 3 In step 304, a flag indicating that the detection result is consistent with the two-way interpolation frame can be output, so that the main and secondary camera interpolation frames are output based on the indication of the flag, that is, Figure 3 The N_t-th frame image and the M_t-th frame image in . Figure 2 In step 205 shown in FIG. 1 , a first video frame and a second video frame are output based on the same relative timestamp.
[0072] Correspondingly, combined Figure 3 In the third video, the Nth frame image, the N_tth frame image, and the N+1th frame image are included in sequence. In the fourth video, the Mth frame image, the M_tth frame image, and the M+1th frame image are included in sequence.
[0073] In this embodiment, a consistency check is performed on two interpolated frames associated with the same timestamp. Specifically, the optical flow generated between the interpolated frame and the adjacent frame is used as a reference to compare the linear correlation of the optical flow generated by the two paths. If the linear correlation is small enough, a frame consistency detection flag is output, indicating that the two-path interpolation is consistent, and the interpolation is completed.
[0074] In the process of the video processing method of another embodiment of the present application, step 120 further includes:
[0075] Sub-step C1: when the linear correlation degree is greater than the degree threshold, fuse the Nth frame image and the Mth frame image to obtain a first fused frame, and fuse the N+1th frame image and the M+1th frame image to obtain a second fused frame.
[0076] In this step, if the linear correlation degree is greater than the degree threshold, the two-way interpolation frames are inconsistent and need to be processed subsequently.
[0077] For reference, when the linear correlation degree is greater than the degree threshold, such as Figure 3 In step 305 shown, a flag indicating that the detection result is that the two-way interpolation frames are inconsistent may be output, so that corresponding processing is performed based on the indication of the flag.
[0078] The specific processing process is as follows: Figure 2 As shown, step 206 corresponds to step C1. The synthesized first video frame remains unchanged, and what needs to be done is to fuse the Nth frame image of the first video with the Mth frame image of the second video, and to fuse the N+1th frame image of the first video with the M+1th frame image of the second video, so as to obtain the first fused frame and the second fused frame respectively.
[0079] Sub-step C2: synthesizing a third fused frame according to the first fused frame and the second fused frame.
[0080] like Figure 2 As shown, step 207 corresponds to step C2.
[0081] In this step, the same frame interpolation algorithm is used to synthesize a third fused frame between the first fused frame and the second fused frame.
[0082] Sub-step C3: Obtain a third video based on the Nth frame image, the first video frame, and the N+1th frame image, and obtain a fourth video based on the Mth frame image, the third fused frame, and the M+1th frame image.
[0083] like Figure 2 As shown, step 208 corresponds to step C3.
[0084] In this step, since the first video frame is inconsistent with the second video frame, after the frame images of the first video and the second video are fused, a frame image is synthesized between the two fused frames based on the fused frame, which is defined as a third fused frame, and the second video frame is replaced by the third fused frame.
[0085] In this embodiment, after the first video frame and the second video frame are synthesized based on the same relative timestamp, if the two frames are inconsistent, the frame images in the first video and the second video are fused to reduce the difference between the two video frame images, and then interpolation processing is performed on the fused frame image to obtain a synthesized frame image, which is used as an interpolation frame in the second video. This can minimize the difference between the interpolation frames in the dual-channel video and ensure the consistency of the interpolation frames.
[0086] In the process of the video processing method of another embodiment of the present application, before step 110, the method further includes:
[0087] Step D1: Capture a first video through a first camera, and capture a second video through a second camera.
[0088] Optionally, a new page is opened in the camera program, and the page is used to display a shooting preview interface of a 3D shooting mode, and the 3D shooting mode is used to shoot 3D videos.
[0089] like Figure 4 The page shown is used to display the shooting preview interface of the 3D shooting mode, so that when the user clicks the shooting option 401, the first camera starts to collect images as the first video, and the second camera starts to collect images as the second video.
[0090] Correspondingly, step 110 includes:
[0091] Sub-step D2: Displaying video identifiers for indicating the first video and the second video.
[0092] In this step, the first video and the second video shot in the 3D shooting mode are stored in a designated album, such as a 3D album.
[0093] Optionally, in the 3D album interface, a video ID list is displayed, where a video ID is used to indicate a main camera and a secondary camera dual-channel video. Figure 5 As shown, multiple video identifiers are displayed, such as video identifier 501.
[0094] Sub-step D3: receiving a first input of a video identifier.
[0095] In some embodiments of the present application, the above-mentioned first input is used for the user to operate the video identifier indicating the first video and the second video to trigger the playback of a high frame rate 3D video, wherein the 3D video is played according to the third video and the fourth video. Therefore, before playing the high frame rate 3D video, the currently stored low frame rate first video and the second video are interpolated. The first input can be a first operation. Exemplarily, the above-mentioned first input includes but is not limited to: the user uses a touch device such as a finger or a stylus to touch the controls on the screen, the touch input to the screen area, or the voice command input by the user, or the specific gesture input by the user, or other feasible input, which can be determined according to actual use requirements, and the embodiment of the present invention is not limited. The specific gesture in the embodiment of the present application can be any one of a single-click gesture, a sliding gesture, a drag gesture, a pressure recognition gesture, a long press gesture, an area change gesture, a double press gesture, and a double-click gesture; the click input in the embodiment of the present application can be a single-click input, a double-click input, or any number of click inputs, etc., and can also be a long press input or a short press input.
[0096] For example, the first input mentioned above is that the user long presses the video identifier used to indicate the first video and the second video, so that a selection box 502 is displayed in the upper right corner of the video identifier. In addition, options for various operations are also displayed, such as option 503 for 3D viewing. The user clicks option 503 for 3D viewing, and clicks selection box 502 in the upper right corner of the video identifier.
[0097] Sub-step D4: In response to the first input, obtain the first video and the second video.
[0098] In this step, the first video and the second video with a low frame rate may be acquired by the system-level chip.
[0099] Furthermore, the system-level chip performs frame insertion processing on the first video and the second video respectively, such as Figure 6 As shown, in the application scenario, the user wears AR glasses 601 and plays a high frame rate 3D video in the AR glasses 601.
[0100] In this embodiment, the user shoots a 3D video on an electronic device, and the video shot is a low-frame-rate video, which is saved in a 3D album. In the playback scenario of the 3D video, the user triggers the interpolation of the low-frame-rate video through input, so that the low-frame-rate video is upgraded to a high-frame-rate video, and finally sent to a display device such as AR glasses for playback. It can be seen that this embodiment only improves the frame rate of a low-frame-rate 3D video in the playback scenario, while ensuring the user's viewing experience, avoiding the increase in power consumption and storage space occupied by the electronic device due to shooting a high-frame-rate 3D video, so as to optimize the performance of the electronic device.
[0101] In summary, the present application proposes a dual-channel video interpolation algorithm, which combines the guidance of timestamps when interpolating dual-channel videos to ensure that the synthesized main and auxiliary camera intermediate frames remain synchronized, that is, to ensure the consistency of interpolation between the main and auxiliary camera channels, and ultimately achieve the purpose of improving the frame rate of dual-camera videos to increase the smoothness of 3D video viewing. At the same time, electronic devices do not need to increase the video frame rate during shooting, and the videos stored in electronic devices are also conventional specifications, such as 1080p and 30fps, which will not burden the power consumption and storage space of electronic devices. Only in specified scenarios, such as when users watch 3D videos through AR glasses, the interpolation algorithm integrated in the external SOC chip increases the video frame rate from 30FPS to 60FPS, which can reduce the load on the application processor (AP) of the electronic device and avoid memory usage caused by storing high frame rate videos in electronic devices.
[0102] The video processing method provided in the embodiment of the present application can be executed by a video processing device. In the embodiment of the present application, the video processing device provided in the embodiment of the present application is described by taking the video processing method executed by the video processing device as an example.
[0103] Figure 7 A block diagram of a video processing device according to an embodiment of the present application is shown, the device comprising:
[0104] An acquisition module 10 is used to acquire a first video captured by a first camera and a second video captured by a second camera;
[0105] The processing module 20 is used to perform frame interpolation processing between two adjacent frames in the first video to obtain a third video, and to perform frame interpolation processing between two adjacent frames in the second video to obtain a fourth video;
[0106] The output module 30 is used to output a three-dimensional video based on the third video and the fourth video.
[0107] In an embodiment of the present application, in a shooting scene, the first camera captures an image to obtain a first video, and the second camera captures an image to obtain a second video. Further, the first video and the second video are obtained, and interpolation processing is performed between adjacent frames of the first video to obtain a third video, and interpolation processing is performed between adjacent frames of the second video to obtain a fourth video. Based on this, the third video is a high frame rate video compared to the first video, and the fourth video is a high frame rate video compared to the second video. Finally, according to the third video and the fourth video, a 3D video is output, so as to achieve the effect of displaying a dual-camera video with a higher frame rate. In particular, in an AR display scene, a 3D video can be displayed in an AR device to ensure the smoothness of the displayed 3D video and enhance the user's viewing experience. It can be seen that the embodiment of the present application can increase the frame rate of the dual-camera video, and in a 3D display scene, the amount of information received by the user's eyes can be increased, so as to better present a smooth 3D effect.
[0108] Optionally, the frame rate of the first video is the same as the frame rate of the second video; the processing module 20 includes:
[0109] A first synthesis unit, configured to synthesize a first video frame according to an Nth frame image and an N+1th frame image of a first video, wherein N is a positive integer;
[0110] A first acquisition unit, configured to acquire a relative timestamp of a first video frame; wherein the relative timestamp of the first video frame is used to indicate that the first video frame is located between an Nth frame image and an N+1th frame image;
[0111] A determination unit, configured to determine the Mth frame image and the M+1th frame image of the second video according to the relative timestamp of the first video frame; wherein M is a positive integer;
[0112] The second synthesis unit is used to synthesize a second video frame according to the Mth frame image and the M+1th frame image.
[0113] Optionally, the processing module 20 further includes:
[0114] A first calculation unit is used to calculate the optical flow between the Nth frame image and the first video frame, and the optical flow between the first video frame and the N+1th frame image, to obtain a first optical flow;
[0115] A second calculation unit is used to calculate the optical flow between the M-th frame image and the second video frame, and the optical flow between the second video frame and the M+1-th frame image, to obtain a second optical flow;
[0116] A third calculation unit, used for calculating the linear correlation degree according to the first optical flow and the second optical flow;
[0117] The first processing unit is used to obtain a third video according to the Nth frame image, the first video frame, and the N+1th frame image when the degree of linear correlation is less than or equal to a degree threshold, and to obtain a fourth video according to the Mth frame image, the second video frame, and the M+1th frame image.
[0118] Optionally, the processing module 20 further includes:
[0119] a fusion unit, configured to fuse the Nth frame image and the Mth frame image to obtain a first fused frame, and to fuse the N+1th frame image and the M+1th frame image to obtain a second fused frame when the linear correlation degree is greater than a degree threshold;
[0120] A third synthesis unit, configured to synthesize a third fused frame according to the first fused frame and the second fused frame;
[0121] The second processing unit is used to obtain a third video according to the Nth frame image, the first video frame, and the N+1th frame image, and to obtain a fourth video according to the Mth frame image, the third fused frame, and the M+1th frame image.
[0122] Optionally, the device further comprises:
[0123] A capture module, used to capture a first video through a first camera, and capture a second video through a second camera;
[0124] Get modules, including:
[0125] A display unit, used to display a video identifier for indicating the first video and the second video;
[0126] A receiving unit, configured to receive a first input of a video identifier;
[0127] The second acquisition unit is used to acquire the first video and the second video in response to the first input.
[0128] The device in the application embodiment can be an electronic device or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc. It can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., and the present application embodiment is not specifically limited.
[0129] The device of the embodiment of the present application may be a device having an action system. The action system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.
[0130] The device provided in the embodiment of the present application can implement each process implemented in the above method embodiment and achieve the same technical effect. To avoid repetition, it will not be repeated here.
[0131] Alternatively, if Figure 8 As shown, an embodiment of the present application also provides an electronic device 100, including a processor 101, a memory 102, and a program or instruction stored in the memory 102 and executable on the processor 101. When the program or instruction is executed by the processor 101, each step of any of the above-mentioned video processing method embodiments is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.
[0132] It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.
[0133] Fig. 9 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.
[0134] The electronic device 1000 includes but is not limited to: a radio frequency unit 1001, a network module 1002, an audio output unit 1003, an input unit 1004, a sensor 1005, a display unit 1006, a user input unit 1007, an interface unit 1008, a memory 1009, a processor 1010, a camera 1011 and other components.
[0135] Those skilled in the art will appreciate that the electronic device 1000 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 1010 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig. 9 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.
[0136] Among them, processor 1010 is used to obtain a first video captured by a first camera and a second video captured by a second camera; perform frame interpolation processing between two adjacent frames in the first video to obtain a third video, and perform frame interpolation processing between two adjacent frames in the second video to obtain a fourth video; and output a three-dimensional video based on the third video and the fourth video.
[0137] In an embodiment of the present application, in a shooting scene, the first camera captures an image to obtain a first video, and the second camera captures an image to obtain a second video. Further, the first video and the second video are obtained, and interpolation processing is performed between adjacent frames of the first video to obtain a third video, and interpolation processing is performed between adjacent frames of the second video to obtain a fourth video. Based on this, the third video is a high frame rate video compared to the first video, and the fourth video is a high frame rate video compared to the second video. Finally, according to the third video and the fourth video, a 3D video is output, so as to achieve the effect of displaying a dual-camera video with a higher frame rate. In particular, in an AR display scene, a 3D video can be displayed in an AR device to ensure the smoothness of the displayed 3D video and enhance the user's viewing experience. It can be seen that the embodiment of the present application can increase the frame rate of the dual-camera video, and in a 3D display scene, the amount of information received by the user's eyes can be increased, so as to better present a smooth 3D effect.
[0138] Optionally, the frame rate of the first video is the same as the frame rate of the second video; the processor 1010 is further used to synthesize a first video frame based on the Nth frame image and the N+1th frame image of the first video, wherein N is a positive integer; obtain a relative timestamp of the first video frame; wherein the relative timestamp of the first video frame is used to characterize that the first video frame is located between the Nth frame image and the N+1th frame image; according to the relative timestamp of the first video frame, determine the Mth frame image and the M+1th frame image of the second video; wherein M is a positive integer; synthesize a second video frame based on the Mth frame image and the M+1th frame image.
[0139] Optionally, the processor 1010 is further used to calculate the optical flow between the Nth frame image and the first video frame, and the optical flow between the first video frame and the N+1th frame image to obtain a first optical flow; calculate the optical flow between the Mth frame image and the second video frame, and the optical flow between the second video frame and the M+1th frame image to obtain a second optical flow; calculate the degree of linear correlation based on the first optical flow and the second optical flow; when the degree of linear correlation is less than or equal to a degree threshold, obtain a third video based on the Nth frame image, the first video frame, and the N+1th frame image, and obtain a fourth video based on the Mth frame image, the second video frame, and the M+1th frame image.
[0140] Optionally, the processor 1010 is further used to, when the degree of linear correlation is greater than the degree threshold, fuse the Nth frame image and the Mth frame image to obtain a first fused frame, and fuse the N+1th frame image and the M+1th frame image to obtain a second fused frame; synthesize a third fused frame based on the first fused frame and the second fused frame; obtain a third video based on the Nth frame image, the first video frame, and the N+1th frame image; and obtain a fourth video based on the Mth frame image, the third fused frame, and the M+1th frame image.
[0141] Optionally, the processor 1010 is also used to capture a first video through a first camera and a second video through a second camera; the display unit 1006 is used to display a video identifier indicating the first video and the second video; the user input unit 1007 is used to receive a first input for the video identifier; the processor 1010 is also used to obtain the first video and the second video in response to the first input.
[0142] In summary, the present application proposes a dual-channel video interpolation algorithm, which combines the guidance of timestamps when interpolating dual-channel videos to ensure that the synthesized main and auxiliary camera intermediate frames remain synchronized, that is, to ensure the consistency of interpolation in the main and auxiliary camera channels, and ultimately achieve the purpose of improving the frame rate of dual-camera videos to increase the smoothness of 3D video viewing. At the same time, electronic devices do not need to increase the video frame rate during shooting, and the videos stored in electronic devices are also conventional specifications, such as 1080p and 30fps, which will not burden the power consumption and storage space of electronic devices. Only in specified scenarios, such as when users watch 3D videos through AR glasses, the interpolation algorithm integrated in the external SOC chip increases the video frame rate from 30FPS to 60FPS, which can reduce the load on the application processor (AP) of the electronic device and avoid memory usage caused by storing high frame rate videos in electronic devices.
[0143] It should be understood that in the embodiment of the present application, the input unit 1004 may include a graphics processor (Graphics Processing Unit, GPU) 10041 and a microphone 10042, and the graphics processor 10041 processes the image data of the static picture or video image obtained by the image capture device (such as a camera) in the video image capture mode or the image capture mode. The display unit 1006 may include a display panel 10061, and the display panel 10061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1007 includes a touch panel 10071 and at least one of other input devices 10072. The touch panel 10071 is also called a touch screen. The touch panel 10071 may include two parts: a touch detection device and a touch controller. Other input devices 10072 may include, but are not limited to, a physical keyboard, a function key (such as a volume control button, a switch button, etc.), a trackball, a mouse, and an action rod, which will not be repeated here. The memory 1009 can be used to store software programs and various data, including but not limited to applications and action systems. The processor 1010 may integrate an application processor and a modem processor, wherein the application processor mainly processes the action system, user pages and application programs, etc., and the modem processor mainly processes wireless communications. It is understandable that the modem processor may not be integrated into the processor 1010.
[0144] The memory 1009 can be used to store software programs and various data. The memory 1009 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 1009 may include a volatile memory or a non-volatile memory, or the memory 1009 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 1009 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.
[0145] The processor 1010 may include one or more processing units; optionally, the processor 1010 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 1010.
[0146] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned video processing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.
[0147] The processor is the processor in the electronic device in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.
[0148] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned video processing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.
[0149] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.
[0150] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned video processing method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.
[0151] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.
[0152] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods of each embodiment of the present application.
[0153] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.
Claims
1. A video processing method, characterized in that: The method comprises: Acquire a first video captured by the first camera and a second video captured by the second camera; Performing frame interpolation between two adjacent frames in the first video to obtain a third video, and performing frame interpolation between two adjacent frames in the second video to obtain a fourth video; Based on the third video and the fourth video, a three-dimensional video is output.
2. The method according to claim 1, characterized in that: The frame rate of the first video is the same as the frame rate of the second video; The interpolating frame processing between two adjacent frames in the first video to obtain a third video, and the interpolating frame processing between two adjacent frames in the second video to obtain a fourth video, include: Synthesize a first video frame according to the Nth frame image and the N+1th frame image of the first video, wherein N is a positive integer; Obtaining a relative timestamp of the first video frame; wherein the relative timestamp of the first video frame is used to indicate that the first video frame is located between the Nth frame image and the N+1th frame image; Determine the Mth frame image and the M+1th frame image of the second video according to the relative timestamp of the first video frame; wherein M is a positive integer; A second video frame is synthesized according to the Mth frame image and the M+1th frame image.
3. The method according to claim 2, characterized in that After synthesizing the second video frame, interpolating between two adjacent frames in the first video to obtain a third video, and interpolating between two adjacent frames in the second video to obtain a fourth video, further comprising: Calculating the optical flow between the Nth frame image and the first video frame, and the optical flow between the first video frame and the N+1th frame image to obtain a first optical flow; Calculating the optical flow between the M-th frame image and the second video frame, and the optical flow between the second video frame and the M+1-th frame image, to obtain a second optical flow; Calculating a linear correlation degree according to the first optical flow and the second optical flow; When the degree of linear correlation is less than or equal to the degree threshold, a third video is obtained based on the Nth frame image, the first video frame, and the N+1th frame image, and a fourth video is obtained based on the Mth frame image, the second video frame, and the M+1th frame image.
4. The method according to claim 3, characterized in that: The interpolation process is performed between two adjacent frames in the first video to obtain a third video, and the interpolation process is performed between two adjacent frames in the second video to obtain a fourth video, further comprising: When the linear correlation degree is greater than the degree threshold, fusing the Nth frame image and the Mth frame image to obtain a first fused frame, and fusing the N+1th frame image and the M+1th frame image to obtain a second fused frame; synthesizing a third fused frame according to the first fused frame and the second fused frame; A third video is obtained according to the Nth frame image, the first video frame, and the N+1th frame image, and a fourth video is obtained according to the Mth frame image, the third fused frame, and the M+1th frame image.
5. The method according to claim 1, characterized in that: Before acquiring the first video captured by the first camera and the second video captured by the second camera, the method further includes: Capturing a first video through a first camera, and capturing a second video through a second camera; The obtaining of the first video captured by the first camera and the second video captured by the second camera includes: displaying video identifiers for indicating the first video and the second video; receiving a first input of the video identifier; In response to the first input, the first video and the second video are acquired.
6. A video processing device, characterized in that: The device comprises: An acquisition module, used to acquire a first video captured by the first camera and a second video captured by the second camera; A processing module, configured to perform frame interpolation processing between two adjacent frames in the first video to obtain a third video, and perform frame interpolation processing between two adjacent frames in the second video to obtain a fourth video; An output module is used to output a three-dimensional video based on the third video and the fourth video.
7. The device according to claim 6, characterized in that The frame rate of the first video is the same as the frame rate of the second video; the processing module includes: A first synthesis unit, configured to synthesize a first video frame according to the Nth frame image and the N+1th frame image of the first video; wherein N is a positive integer; A first acquisition unit, configured to acquire a relative timestamp of the first video frame; wherein the relative timestamp of the first video frame is used to indicate that the first video frame is located between the Nth frame image and the N+1th frame image; A determining unit, configured to determine the Mth frame image and the M+1th frame image of the second video according to the relative timestamp of the first video frame; wherein M is a positive integer; The second synthesis unit is used to synthesize a second video frame according to the Mth frame image and the M+1th frame image.
8. The device according to claim 7, characterized in that The processing module further includes: A first calculation unit, configured to calculate an optical flow between the Nth frame image and the first video frame, and an optical flow between the first video frame and the N+1th frame image, to obtain a first optical flow; A second calculation unit is used to calculate the optical flow between the M-th frame image and the second video frame, and the optical flow between the second video frame and the M+1-th frame image, to obtain a second optical flow; A third calculation unit, configured to calculate a linear correlation degree according to the first optical flow and the second optical flow; The first processing unit is used to obtain a third video according to the Nth frame image, the first video frame, and the N+1th frame image when the linear correlation degree is less than or equal to the degree threshold, and to obtain a fourth video according to the Mth frame image, the second video frame, and the M+1th frame image.
9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the video processing method according to any one of claims 1 to 5 are implemented.
10. A readable storage medium, characterized in that: The readable storage medium stores a program or an instruction, and when the program or the instruction is executed by the processor, the steps of the video processing method according to any one of claims 1 to 5 are implemented.
Citation Information
Patent Citations
Video processing method and device, electronic equipment and readable storage medium
CN115049574A
Binocular video processing method and device, equipment and storage medium
CN118337970A
