Video processing method, electronic equipment, chip system and storage medium

Abstract

The invention discloses a video processing method, electronic equipment, a chip system and a storage medium, relates to the technical field of computer vision, and can automatically shoot an ascending video with relatively high image quality. The method is applied to the electronic equipment, the electronic equipment can control cameras with different frame rates to shoot videos at the same time, and for the same scene, after video clips corresponding to the scene in a high-frame-rate video are processed at a low speed, video clips corresponding to the scene in a low-frame-rate video are replaced, so that an ascending video is generated.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer vision technology, and in particular to a video processing method, electronic device, chip system and storage medium. Background Art

[0002] In today's electronic society, people are accustomed to using electronic devices to shoot videos for record keeping. For example, using electronic devices to shoot videos to record important scenes or important moments.

[0003] With the further development of computer vision technology, the video recording function of electronic devices is becoming more and more optimized. People use electronic devices to shoot videos not only to simply record events, but also to use electronic devices for more professional shooting. For example, electronic devices are used to shoot slow-motion videos (i.e. videos with slow motion effects).

[0004] Some traditional methods of shooting upscaled videos first use a low frame rate camera to shoot the video, and then use interpolation to achieve the upscaled video effect. However, the upscaled video effect achieved by this interpolation method often results in low video clarity and poor quality of the upscaled video. Summary of the invention

[0005] The embodiments of the present application provide a video processing method, an electronic device, a chip system and a storage medium, which can automatically shoot a high-quality upscaling video compared to the upscaling video generated by interpolation.

[0006] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0007] In the first aspect, the present application provides a video processing method, which is applied to an electronic device, wherein the electronic device includes a first camera and a second camera; the lenses of the first camera and the second camera are facing the same direction. The electronic device can control the first camera and the second camera to shoot videos simultaneously using different frame rates, so as to obtain a first video corresponding to the first frame rate and a second video corresponding to the second frame rate, wherein the first frame rate is lower than the second frame rate. After shooting, a third video can be saved, and the third video is a video obtained by replacing the first video clip in the first video with the third video clip; the third video clip is obtained by slow processing the second video clip in the second video; the second video clip and the first video clip correspond to the same time period on the timeline, which is equivalent to corresponding to the same scene. That is, the same scene corresponds to the first video clip and the second video clip in the first video and the second video clip, respectively.

[0008] In the above scheme, cameras with different frame rates are controlled to shoot videos at the same time. For the same scene, the video clip corresponding to the scene in the high frame rate video is slowed down, which can extend the playback time of the scene. Then, the video clip corresponding to the scene in the low frame rate video is replaced with the slowed-down video clip, so that the video (i.e., the third video) can be automatically upscaled. The video frame corresponding to the scene in the upscaled video is the upscaled frame. Since the upscaled frame is the video frame actually captured by the camera, the upscaled frame has a higher picture clarity than the interpolated frame, thereby improving the picture quality of the upscaled video.

[0009] In addition, the solution of the present application can not only automatically generate high-quality upscaling videos, but also can specifically upscale a video clip without slowing down the entire video, which is more flexible and more applicable. Moreover, the upscaling video finally generated is also a low frame rate video, which is equivalent to achieving the upscaling effect in the subsequent low frame rate playback, greatly reducing the requirements for the hardware performance of the electronic equipment and improving the stability.

[0010] In a possible implementation manner of the first aspect, a field of view of the first camera is greater than a field of view of the second camera; and a picture area of ​​the same object in the third video clip is greater than a picture area in the first video clip.

[0011] In the above solution, since the picture area in the third video clip is directly captured by the second camera, it does not need to undergo additional zooming processing. After the upscaled video is played to the third video clip, the upscaled + zooming effect can be automatically achieved, avoiding the problem of deterioration of image clarity caused by additional zooming processing of the image, thereby obtaining a high-quality automatically upscaled and zoomed video, improving the user experience. In addition, no additional automatic zooming processing is required, which can also save computing processing resources.

[0012] In a possible implementation of the first aspect, a picture preview area of ​​a shooting interface of the electronic device can display a picture captured by the first camera in real time using a first frame rate, and a field of view of the second camera can be indicated in the picture preview area, that is, field of view indication information is displayed. Based on the shooting interface, the first camera and the second camera can be triggered to simultaneously control the shooting of a video.

[0013] In the above scheme, the field of view is marked in the picture preview area of ​​the shooting interface, so that the user can understand the field of view of the second camera when shooting and adjust the shooting angle, so that the pictures taken by the first camera and the second camera are more accurate, which is convenient for the subsequent automatic generation of better and more complete upscaling videos.

[0014] In a possible implementation of the first aspect, the shooting interface includes an entry to a target recording mode; the target recording mode is a specific recording mode for recording an automatically upscaled video. When the entry is triggered, the target recording mode can be entered or started, and field of view indication information is displayed in a screen preview area of ​​the shooting interface.

[0015] In the above scheme, a dedicated recording mode (i.e., target recording mode) is proposed. After entering the dedicated recording mode, the real-time image captured by the camera with a large field of view angle can be previewed in the shooting interface, and the field of view of the camera with a smaller field of view angle can be marked at the same time, so as to facilitate subsequent more accurate video shooting and generate more accurate upscaling videos.

[0016] In a possible implementation of the first aspect, the first video is shot when the first camera is 3A controlled by using first 3A control information; the second video is shot when the second camera is 3A controlled by referring to the first 3A control information corresponding to the first camera.

[0017] In the above scheme, the 3A control of the high frame rate camera can refer to the 3A control information of the low frame rate camera, so that the image effects captured by the two cameras are closer, so that when the subsequently spliced ​​up-motion video is played to the up-motion segment (i.e., the third video segment), the screen switching transition is more natural. From the user's visual perception level, it looks like the automatic up-motion effect achieved by one camera.

[0018] In a possible implementation of the first aspect, the first 3A control information is calculated based on the first image statistical parameters corresponding to the video frame collected by the first camera. The electronic device can control the second camera to shoot the video using the second frame rate, and perform 3A control based on the third 3A control information during the video shooting process to obtain the shot second video; wherein the third 3A control information is closer to the first 3A control information than the second 3A control information; and the second 3A control information is calculated based on the second image statistical parameters corresponding to the video frame collected by the second camera.

[0019] In the above scheme, the electronic device can calculate the corresponding 3A control information according to the image statistical parameters of the image collected by each camera, and then can jointly determine the 3A control information ultimately used to control the high frame rate camera (i.e., the second camera) according to the corresponding 3A control information calculated by the image statistical parameters. Since the 3A control information ultimately used to control the high frame rate camera refers to the statistical characteristics of the image parameters of the images collected by the two cameras, it can maintain close to the 3A control of the low frame rate camera, and can also take into account the 3A control requirements of the high frame rate camera itself, and is more applicable and more accurate 3A control information.

[0020] In a possible implementation of the first aspect, the electronic device includes a first 3A module and a second 3A module; the first 3A control information is calculated by the first 3A module based on the first image statistical parameters; the second 3A control information is calculated by the second 3A module based on the second image statistical parameters. The electronic device can synchronize the first 3A control information calculated by the first 3A module to the second 3A module; and determine the third 3A control information through the second 3A module according to the second 3A control information and the first 3A control information synchronized to it.

[0021] In the above scheme, the electronic device gives the 3A module a 3A synchronization function. After the 3A control modules of each camera calculate the 3A control information, the 3A control information can be synchronized by the 3A module, which is more convenient and efficient.

[0022] In a possible implementation of the first aspect, the first video segment is a highlight segment in the first video; the highlight segment corresponds to a target time period on the timeline of the first video; and the second video segment is a video segment in the second video that is within the target time period.

[0023] In the above solution, the electronic device can automatically upscale the highlight segments in the captured video in a targeted manner, which is more flexible. Moreover, in general, the highlight segments are the video segments that better meet the user's upscaling requirements, so it is possible to more accurately generate upscaled videos that meet the user's requirements. In addition, there is no need to upscale the entire video, which greatly saves system resources.

[0024] In a possible implementation of the first aspect, the first video has metadata carrying a highlight action tag; the highlight action tag in the metadata has a corresponding timestamp; and the target time period is a time period obtained by expanding a preset time range based on the timestamp.

[0025] In the above scheme, by marking metadata with highlight action tags on the timeline, there is no need for complex analysis and processing, that is, it is possible to more conveniently and accurately capture highlight clips from videos shot at different frame rates for upscaling processing, and to more quickly and accurately stitch videos to form upscaling videos.

[0026] In a possible implementation of the first aspect, the electronic device may perform highlight action recognition processing based on the video frames captured by the first camera; add a highlight action tag corresponding to the timestamp of the video frame in which the highlight action is recognized, and obtain metadata of the first video carrying the highlight action tag.

[0027] In the above scheme, video frames with highlight actions can be automatically identified, and highlight action tags can be added corresponding to their timestamps to obtain metadata. Subsequently, based on the metadata, automatic upscaling of highlight clips can be quickly achieved.

[0028] In a possible implementation of the first aspect, the electronic device may discard the first video segment in the first video; insert the third video segment into a target position in the first video, and splice the inserted third video segment with the remaining video frames in the first video to obtain the third video; the target position is the position of the first video segment in the first video.

[0029] In the above scheme, by splicing the third video segment after slow motion processing of the high frame rate video segment with the remaining video frames in the low frame rate video in an orderly manner according to reasonable positions, a smooth playback, normal timeline and high picture quality upscaling video can be obtained.

[0030] In a second aspect, the present application provides an electronic device, which includes at least: a first camera and a second camera, a memory, and one or more processors; the first camera and the second camera are used to capture images, and the memory is used to store computer instructions. When the one or more processors execute the computer instructions, the electronic device executes a method as described in any one of the above-mentioned first aspects.

[0031] In a third aspect, the present application provides a chip system, which is applied to an electronic device, and the chip system includes one or more processors, which are used to call computer instructions so that the electronic device executes any one of the methods of the first aspect above.

[0032] In a fourth aspect, the present application provides a computer storage medium, which includes computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes any one of the methods of the first aspect.

[0033] In a fifth aspect, the present application provides a computer program product, which, when executed on a computer, enables the computer to execute any one of the methods in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A schematic diagram of extending the playback time provided in an embodiment of the present application;

[0035] Figure 2 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0036] Figure 3 A software structure block diagram of an electronic device provided in an embodiment of the present application;

[0037] FIG. 4A to FIG. 4B A schematic diagram of an automatic magnification effect provided in an embodiment of the present application;

[0038] Figure 4C A schematic diagram of a multi-step upscaling amplification effect provided by an embodiment of the present application;

[0039] Figure 5 A schematic diagram of a shooting interface provided in an embodiment of the present application;

[0040] Figure 6 A schematic diagram of an interface for starting a target video recording mode provided in an embodiment of the present application;

[0041] Figure 7 A schematic diagram of a first video clip and a second video clip provided in an embodiment of the present application;

[0042] Figure 8 A schematic diagram of splicing and generating upscaled video provided in an embodiment of the present application;

[0043] Fig. 9 A schematic diagram of a video processing method provided in an embodiment of the present application;

[0044] Fig.10 A schematic diagram of the principle of a video shooting process provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] References to "one embodiment" or "some embodiments" etc. described in this specification mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Thus, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways. The term "connected" includes direct and indirect connections, unless otherwise stated.

[0046] In the embodiments of the present application, the words "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0047] Before formally introducing the solutions of the embodiments of the present application, it should be emphasized that, hereinafter, the terms "first", "second" and "third" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first", "second" and "third" may explicitly or implicitly include one or more of the features. In the description of this embodiment, unless otherwise specified, "multiple" means two or more.

[0048] The embodiment of the present application provides a video processing method. The method is used to automatically shoot a scaled-up video through an electronic device. A scaled-up video refers to a video with a scaled-up effect (or a slow-motion effect). The scaled-up frame of the video with a scaled-up effect generated by the method in the embodiment of the present application has a higher definition, which overcomes the problem of low picture clarity of the scaled-up frame in the traditional method. The scaled-up frame is a video frame (also referred to as an image frame) in the video with a slow-motion effect or a scaled-up effect.

[0049] To facilitate understanding, some of the knowledge involved in the embodiments of this application is now introduced.

[0050] Slow motion is a filming technique that is usually used to achieve a slow motion effect.

[0051] Specifically, when the video shooting frame rate and the electronic device playback frame rate are the same, it can give users a natural and smooth video viewing experience. If the video shooting frame rate is higher than the electronic device playback frame rate, it will present a slow motion effect, which is called upscaling.

[0052] The shooting frame rate is the frame rate used by the camera when shooting video, that is, the number of images captured by the electronic device in a unit of time. The playback frame rate is the frame rate used by the electronic device when playing video, that is, the number of images played by the electronic device in a unit of time. For example, if the frame rate (i.e., the playback frame rate) of the electronic device when playing video is 30FPS (Frames Per Second, the number of frames transmitted per second), and the frame rate (i.e., the shooting frame rate) when shooting video is 60FPS, a slow motion effect will be produced, which is called upscaling.

[0053] In some solutions, the electronic device will control the camera to shoot video at a low frame rate, so the frame rate of the shot video is relatively low, so the upscaling effect will be achieved through frame insertion. For example, the camera shoots video at a shooting frame rate of 30FPS. If you want to achieve the upscaling effect, you can insert frames through Memc (Motion Estimate and Motion Compensation) to increase the frame rate of the video from 30FPS to 60FPS, and the electronic device will play the video at a playback frame rate of 30FPS. This frame insertion method is worse than the upscaling effect achieved by shooting the video at a real 60FPS frame rate, which will result in a lower picture clarity of the video.

[0054] The embodiment of the present application proposes an improvement plan for the above-mentioned problem. The video processing method in the embodiment of the present application is applied to an electronic device with at least two cameras, and the at least two cameras correspond to different frame rates (i.e., shooting frame rates). The electronic device can respond to the user's video shooting operation and control at least two cameras to collaboratively shoot upscaled videos at different frame rates.

[0055] Specifically, at least two cameras arranged in the electronic device are close to each other and face the same direction. When the electronic device controls at least two cameras to shoot videos at different frame rates at the same time, each camera starts shooting and stops shooting at the same time. Therefore, the videos or video clips shot by different cameras in the same time period all have the same scene, which can be called a shared scene. A shared scene refers to a scene that appears in videos with different frame rates (i.e., videos shot by different cameras at different frame rates) in the same time period.

[0056] If you want to achieve an upscaling effect for a common scene, the electronic device can slow down the video clip of the common scene in the high frame rate video to extend its playback time, and then replace the video clip of the common scene in the low frame rate video with the slowed-down video clip to generate a new video.

[0057] It can be understood that after obtaining a new video in the above manner, when playing the shared scene in the new video, the upscaling effect can be achieved due to the extended playback time. In the new video, the video frames in the video clip corresponding to the shared scene are upscaling frames. Since the upscaling frames are video frames actually captured by the camera, compared with the interpolated frames, the upscaling frames have higher picture clarity and better effects. It should be noted that the above-mentioned "high frame rate video" and "low frame rate video" are relative, that is, in videos with different frame rates, videos with relatively high frame rates are high frame rate videos, and videos with relatively low frame rates are low frame rate videos.

[0058] For ease of understanding, an example is given. For example, an electronic device controls the first camera and the second camera to shoot a video at the same time, wherein the first camera shoots the first video at a frame rate of 30FPS, and the second camera shoots the second video at a frame rate of 60FPS. Then, relatively speaking, the first video is a low frame rate video, and the second video is a high frame rate video. Assume that the first video clip in the first video is used to present scene A, and the first video clip corresponds to the 5th to 6th seconds in the first video, and the second video clip also presents scene A from the 5th to the 6th seconds in the second video, and scene A is the shared scene.

[0059] If you need to automatically achieve a high-speed effect for scene A, you can slow down the second video clip by 2 times, so as to achieve the effect of extending the playback time by 2 times. For example, the frame rate of the second video clip is reduced from 60FPS to 30FPS to obtain the third video clip. Since the second video clip corresponds to the 5th to 6th seconds in the second video, the original playback time of the second video clip is 2s, and the number of video frames in the second video clip is 60FPS*2s=120 video frames. After reducing its frame rate to 30FPS, the playback time = 120 video frames / 30FPS = 4s, that is, the playback time of the third video clip is extended by 2 times.

[0060] In order to more intuitively understand the effect of extending the playback time, we now combine Figure 1 Provide a schematic illustration.

[0061] See also Figure 1, the time length of the first video and the second video are consistent, that is, the playback time of the first video and the second video is consistent, and the second frame rate of the second video is twice the first frame rate of the first video, so the inter-frame interval of the first video is greater than the inter-frame interval of the second video. Assuming that the playback time of the first video segment to be upscaled in the first video is t seconds, and the number of video frames in the first video segment is N, then the corresponding second video segment in the second video is also t seconds, and the number of video frames in the second video segment is 2N. The second video segment is slowed down to make its frame rate become the first frame rate to obtain a third video segment. Figure 1 As shown, the number of video frames in the third video clip is still 2N, but since the frame rate of the third video clip is the first frame rate, which is equivalent to the frame interval being increased to match the frame interval of the first video, the playback time is extended from the original t seconds to 2t seconds, that is, the playback time is extended by 2 times.

[0062] Therefore, after the first video segment in the first video is replaced with the third video segment to obtain a new video, when the new video is played to the third video segment, a slow playback effect (i.e., an up-motion effect) of the shared scene A will be presented. The video frame presenting the shared scene A in the new video is an up-motion frame. Since the up-motion frame is actually acquired using a high frame rate of 60 FPS, the up-motion frame has a higher picture clarity and a better effect than inserting frames to 60 FPS.

[0063] In addition, for ease of understanding, a more detailed explanation of "shared scenarios" is given below.

[0064] Since the electronic device controls at least two cameras to shoot videos at different frame rates at the same time, the same scene shot by different cameras corresponds to the same time scale. That is, if at least two cameras shoot the same scene at the same time, then in the videos shot by different cameras at different frame rates (which can be referred to as "videos with different frame rates"), the scene will appear in the same time period. Therefore, if a scene appears in videos with different frame rates in the same time period, the scene can be called a shared scene.

[0065] For example, the electronic device uses the first camera and the second camera to simultaneously shoot a video of a basketball game. Assuming that player 1 performs a shooting action a at the 5th second, then in the first video shot by the first camera at a frame rate of 30 FPS, the starting time of the shooting action a is the 5th second. Similarly, in the second video shot by the second camera at a frame rate of 60 FPS, the starting time of the shooting action a is also the 5th second.

[0066] The scene of "Player 1 doing shooting action a" is the common scene A. If the starting time of shooting action a is the 5th second (for example, the player starts aiming at the basketball hoop with the hand at the 5th second) and the ending time of shooting action a is the 7th second (for example, the player puts down the hand after the shooting is completed at the 7th second), then the time period corresponding to the common scene A is from the 5th second to the 7th second, a total of 3 seconds.

[0067] It should be understood that the video clip of a scene in a video refers to the clip in the video used to present the scene. The video clip of a common scene in a high frame rate video or a low frame rate video refers to the clip in the high frame rate video or the low frame rate video used to present the common scene. In conjunction with the above example, the video clip corresponding to the common scene A in the first video (recorded as the first video clip) is the video clip in the first video presenting "Player 1 doing shooting action a", and the video clip corresponding to the common scene A in the second video (recorded as the second video clip) is the video clip in the second video presenting "Player 1 doing shooting action a".

[0068] In some embodiments, the common scenes may include dynamic scenes. A dynamic scene refers to a scene in which the picture will produce obvious dynamic changes over time. A dynamic scene may include a scene in which an action is produced or a scene in which a scene element changes position. For example, walking, running, jumping, throwing, looking back, etc. are scenes in which an action is produced, and the wind blowing the leaves, etc. are scenes in which a scene element changes position.

[0069] In other embodiments, the shared scene may also include a static scene. For example, a mother looking tenderly at her sleeping baby is a static scene. Although the mother or the baby does not make obvious dynamic movements, the mother's facial expression still changes slightly. If the static scene is upgraded in the video by the video processing method mentioned in the embodiment of the present application, the tender expression of the mother looking at the baby can be presented more clearly, so as to better record the touching moments of family affection.

[0070] The video processing method provided in the embodiment of the present application can be applied to an electronic device having at least two cameras. For example, the electronic device can be a mobile phone, a sports camera (GoPro), a digital camera, a tablet computer, a desktop, a laptop, a handheld computer, a notebook computer, a vehicle-mounted device, a smart home device (such as a smart TV, a smart screen, a large screen, a smart speaker, etc.), an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook, and a cellular phone, a personal digital assistant (personal digital assistant, PDA), an augmented reality (augmented reality, AR)\virtual reality (virtual reality, VR) device, etc. The embodiment of the present application does not impose any special restrictions on the specific form of the electronic device.

[0071] Now combined Figure 2 The internal structure of the electronic device 200 will be described in more detail.

[0072] The electronic device 200 may include a processor 210, an external memory interface 220, an internal memory 221, a universal serial bus (USB) interface 230, a charging management module 240, a power management module 241, a battery 242, an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, an earphone interface 270D, a sensor module 280, a button 290, a motor 291, an indicator 292, a camera 293, a display screen 294, and a subscriber identification module (SIM) card interface 295, etc. The sensor module 280 may include a pressure sensor 280A, a gyroscope sensor 280B, an air pressure sensor 280C, a magnetic sensor 280D, an acceleration sensor 280E, a distance sensor 280F, a proximity light sensor 280G, a fingerprint sensor 280H, a temperature sensor 280J, a touch sensor 280K, an ambient light sensor 280L, a bone conduction sensor 280M, etc.

[0073] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device 200. In other embodiments of the present application, the electronic device 200 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0074] The processor 210 may include one or more processing units, for example, the processor 210 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0075] The controller may be the nerve center and command center of the electronic device 200. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0076] The processor 210 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 210 is a cache memory. The memory may store instructions or data that the processor 210 has just used or cyclically used. If the processor 210 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 210, and thus improves the efficiency of the system.

[0077] In some embodiments, the processor 210 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0078] The electronic device 200 implements the display function through a GPU, a display screen 294, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 294 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 210 may include one or more GPUs, which execute program instructions to generate or change display information.

[0079] The display screen 294 is used to display images, videos, etc. The display screen 294 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 200 may include 1 or N display screens 294, where N is a positive integer greater than 1.

[0080] The electronic device 200 can realize the shooting function through the ISP, the camera 293, the video codec, the GPU, the display screen 294 and the application processor. For example, in each embodiment of the present application, the first video, the second video and the third video with the upscaling effect are captured according to the above-mentioned elements or devices for realizing the shooting function.

[0081] ISP is used to process the data fed back by camera 293. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element (i.e., sensor) through the lens, and the light signal is converted into an electrical signal. The camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 293.

[0082] The camera 293 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 200 may include 1 or N cameras 293, where N is a positive integer greater than 1.

[0083] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 200 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0084] Video codecs are used to compress or decompress digital videos. The electronic device 200 may support one or more video codecs. Thus, the electronic device 200 may play or record videos in a variety of coding formats, such as moving picture experts group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0085] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of the electronic device 200 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0086] The external memory interface 220 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 200. The external memory card communicates with the processor 210 through the external memory interface 220 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0087] The internal memory 221 can be used to store computer executable program codes, which include instructions. The processor 210 executes various functional applications and data processing of the electronic device 200 by running the instructions stored in the internal memory 221. The internal memory 221 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 200 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 221 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0088] The electronic device 200 can implement audio functions such as music playing and recording through the audio module 270, the speaker 270A, the receiver 270B, the microphone 270C, the headphone jack 270D, and the application processor.

[0089] The audio module 270 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signal. The audio module 270 can also be used to encode and decode audio signals. In some embodiments, the audio module 270 can be arranged in the processor 210, or some functional modules of the audio module 270 can be arranged in the processor 210.

[0090] The speaker 270A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The electronic device 200 can listen to music or listen to a hands-free call through the speaker 270A.

[0091] The receiver 270B, also called a "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 200 receives a call or voice message, the voice can be received by placing the receiver 270B close to the human ear.

[0092] Microphone 270C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by approaching the microphone 270C with his mouth to input the sound signal into the microphone 270C. The electronic device 200 can be provided with at least one microphone 270C.

[0093] The earphone interface 270D is used to connect a wired earphone and can be a USB interface 230 or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0094] The pressure sensor 280A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 280A can be disposed on the display screen 294. There are many types of pressure sensors 280A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc.

[0095] The gyro sensor 280B can be used to determine the motion posture of the electronic device 200. In some embodiments, the angular velocity of the electronic device 200 around three axes (i.e., x, y, and z axes) can be determined by the gyro sensor 280B. The gyro sensor 280B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyro sensor 280B detects the angle of the electronic device 200 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the electronic device 200 through reverse movement to achieve anti-shake. The gyro sensor 280B can also be used for navigation and somatosensory game scenes.

[0096] The acceleration sensor 280E can detect the magnitude of the acceleration of the electronic device 200 in all directions (generally three axes). When the electronic device 200 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the electronic device and is applied to applications such as horizontal and vertical screen switching and pedometers.

[0097] The distance sensor 280F is used to measure the distance. The electronic device 200 can measure the distance by infrared or laser. In some embodiments, when shooting a scene, the electronic device 200 can use the distance sensor 280F to measure the distance to achieve fast focusing.

[0098] The ambient light sensor 280L is used to sense the ambient light brightness. The electronic device 200 can adaptively adjust the brightness of the display screen 294 according to the perceived ambient light brightness. The ambient light sensor 280L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 280L can also cooperate with the proximity light sensor 280G to detect whether the electronic device 200 is in a pocket to prevent accidental touch.

[0099] The fingerprint sensor 280H is used to collect fingerprints. The electronic device 200 can use the collected fingerprint characteristics to achieve fingerprint unlocking, access application locks, fingerprint photography, fingerprint answering calls, etc.

[0100] The touch sensor 280K is also called a "touch panel". The touch sensor 280K can be set on the display screen 294, and the touch sensor 280K and the display screen 294 form a touch screen, also called a "touch screen". The touch sensor 280K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 294. In other embodiments, the touch sensor 280K can also be set on the surface of the electronic device 200, which is different from the position of the display screen 294.

[0101] Figure 3 It is a software structure block diagram of the tablet provided in the embodiment of the present application.

[0102] It can be understood that the layered architecture of the Android system can divide the software into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. Figure 3 As shown, the Android system may include four layers, which are, from top to bottom, the application layer, the application framework layer (Framework layer), the hardware abstract layer (HAL) layer, and the kernel layer (Kernel, also called the driver layer).

[0103] The application layer can include a series of application packages, which can include applications such as camera and gallery.

[0104] The application framework layer provides an application programming interface (API) and a programming framework for the application programs in the application layer. The application framework layer includes some predefined functions.

[0105] In some embodiments, the application framework layer may include a camera access interface, wherein the camera access interface may include an access interface for camera management and camera devices. The camera access interface is used to provide an application programming interface and a programming framework for camera applications.

[0106] The hardware abstraction layer is an interface layer located between the application framework layer and the kernel layer, providing a virtual hardware platform for the operating system.

[0107] In an embodiment of the present application, the hardware abstraction layer may include a camera hardware abstraction layer. The camera hardware abstraction layer includes a camera algorithm library. Among them, the camera hardware abstraction layer can provide virtual hardware for camera device 1 (first camera) and camera device 2 (second camera). It can also obtain posture data and transmit it to the camera algorithm library. The camera hardware abstraction layer can also be used to calculate the number of images to be spliced ​​N, and obtain information from the camera algorithm library.

[0108] The camera algorithm library may include an algorithm module and a perception engine. The algorithm module includes several algorithms for processing images, which can be used to realize the splicing of N frames of images to be spliced ​​and other processing. For example, the 3A algorithm can be used for 3A control during the shooting process. The perception engine is provided with a wonderful moment recognition algorithm, which is used to identify whether there are wonderful moments or highlight actions that need to be upgraded in the captured image (i.e., video frame).

[0109] The kernel layer, also known as the driver layer, is the layer between hardware and software. The kernel layer includes drivers for various hardware.

[0110] In some embodiments, the kernel layer may include at least a camera device driver, a digital signal processor driver, and an image processor driver.

[0111] The camera device driver is used to drive the camera sensor to collect images and drive the image signal processor to pre-process the images.

[0112] The digital signal processor driver is used to drive the digital signal processor to process images.

[0113] The image processor driver is used to drive the graphics processor to process images.

[0114] For ease of understanding, an electronic device including two cameras (i.e., a first camera and a second camera) with lenses facing each other is taken as an example. Figure 3 The software system structure in the embodiment of the present application is described as follows:

[0115] 1. The electronic device starts a first camera and a second camera to shoot videos at different frame rates to obtain an image stream and an audio stream.

[0116] In response to the user's video shooting operation, the camera application calls the camera access interface of the application framework layer, starts the camera application, and then sends an instruction to start shooting video by calling camera device 1 (first camera) and camera device 2 (second camera) in the camera hardware abstraction layer. The camera hardware abstraction layer sends the instruction to the camera device driver of the kernel layer, and the camera device driver can start the sensor, such as the sensor of the first camera and the sensor of the first camera. The image light signal is collected by the sensor of the first camera and the sensor of the first camera at the same time, and the image light signals collected by the sensor of the first camera and the sensor of the first camera are transmitted to the image signal processor (ISP, image signal processor) for processing to obtain the original image stream (at least 2 frames of original image frames constitute an image sequence or a video frame sequence). It can be understood that the first camera and the second camera each correspond to one original image stream.

[0117] Exemplarily, in an embodiment of the present application, the image signal processor may include an ISP front end and an ISP back end, and the ISP front end may include two ISP online front ends and one ISP offline front end. The two ISP online front ends are used to process image streams (i.e., video frame sequences) in real time, and each ISP online front end is responsible for one image stream (i.e., processing the image stream of one camera respectively). Specifically, the ISP online front end mainly uses basic algorithms for image preprocessing, such as bad pixel removal, collecting image statistical information, etc. The ISP offline front end performs more complex image processing such as demosaic color interpolation and denoising based on the data output by the ISP online front end.

[0118] The ISP backend is mainly used to use algorithms to perform related processing to enhance image quality based on the images output by the ISP offline frontend. For example, the ISP backend can be used for further image denoising, color enhancement, image sharpening and other processing that can enhance image quality. It can be understood that the data stream collected by the camera is processed by the ISP online frontend, ISP offline frontend and ISP backend in sequence to obtain the original image stream.

[0119] In addition, the camera application also sends a command to start recording through the audio input unit in the audio hardware abstraction layer. The audio hardware abstraction layer sends the command to the audio driver in the kernel layer. The audio driver can start the microphone to collect audio signals and obtain an audio stream.

[0120] Then, the original image stream is transmitted to the camera hardware abstraction layer through the camera device driver. The camera hardware abstraction layer can send the original image stream to the camera algorithm library. Based on the support of the digital signal processor and the image processor, the camera algorithm library can first process the original image stream to obtain the processed stream.

[0121] Exemplarily, during the video shooting process, the ISP (eg, ISP backend) may transmit the original image stream of the first camera to the camera application layer by layer, so as to preview and display the image stream captured in real time by the first camera in the shooting interface of the camera application.

[0122] Exemplarily, in an embodiment of the present application, a 3A algorithm library is also provided in the camera algorithm library. During video shooting, the ISP can call the 3A algorithm library in the camera algorithm library to perform 3A control on the first camera and the second camera, so that the image quality captured by the two cameras is close. In some embodiments of the present application, 3A synchronous control is also implemented based on the 3A algorithm library, that is, the 3A control information of the first camera can be referenced to generate the 3A control information that is ultimately used to perform 3A control on the second camera.

[0123] 2. The electronic device identifies video frames with highlight actions or exciting moments in the low frame rate image stream.

[0124] Based on the support of digital signal processors and image processors, the perception engine in the camera algorithm library can also be called to use the wonderful moment recognition algorithm to detect whether there are wonderful moments (such as highlight actions) in each frame of the image stream (that is, each video frame).

[0125] 3. The electronic device mixes the image stream and the audio stream into original videos, namely, the first video (the original video with a low frame rate shot by the first camera) and the second video (the original video with a high frame rate shot by the second camera).

[0126] In addition, based on the support of digital signal processors and image processors, the image stream and audio stream can be mixed into the original video based on the same time track.

[0127] 4. The electronic device determines a highlight segment in the first video based on the video frames with highlight action in the first video, captures a video segment corresponding to the highlight segment from the second video, and performs slow motion processing on the video segment in the second video to obtain a highlight segment after slow motion processing.

[0128] The camera algorithm library can call the editing algorithm to extract a video segment corresponding to the highlight segment in time from the second video captured by the second camera based on the position of the highlight segment. The video segment extracted from the second video also contains a wonderful moment or a highlight action. The camera algorithm library can call an algorithm to slow down the video segment extracted from the second video to extend its playback time.

[0129] 5. The electronic device replaces the highlight segment in the first video with the highlight segment after slow motion processing to generate a slow-motion video.

[0130] Specifically, the stitching algorithm in the camera algorithm library can be called to replace the highlight segment in the first video with the video segment captured from the second video to stitch together a scaled-up video (i.e., a new video with the wonderful moment automatically scaled up).

[0131] 6. Electronic devices save original videos and upscaled videos.

[0132] The electronic device can save the first video, the second video, and the upscaled video to a gallery application in the application layer.

[0133] In conjunction with the accompanying drawings, the video processing method provided in the embodiments of the present application is described in detail by taking a mobile phone as an example. It should be noted that the video processing method in each embodiment of the present application is not limited to being applied to a mobile phone, but can also be applied to other electronic devices. The following is only illustrated by taking a mobile phone as an example.

[0134] The mobile phone controls at least two cameras with the same lens orientation, and simultaneously shoots videos at different frame rates to obtain multiple videos with different frame rates. After slowing down the video clips in the high frame rate video (video with a relatively high frame rate), the video clips corresponding to the time in the low frame rate video (video with a relatively low frame rate) are replaced to generate a new video. When the new video plays the replaced video clip, the upscaling effect can be achieved.

[0135] Take the case where two cameras (i.e., the first camera and the second camera) are provided in a mobile phone as an example for explanation. The first camera corresponds to the first frame rate, and the second camera corresponds to the second frame rate. The first frame rate and the second frame rate are frame rates used when shooting videos (shooting frame rates), wherein the first frame rate is lower than the second frame rate. The first camera uses the first frame rate to shoot the first video, and the second camera uses the second frame rate to shoot the second video. Then, the frame rate of the first video is lower than the frame rate of the second video. Exemplarily, the above two cameras can be rear cameras in a mobile phone.

[0136] If you want to achieve a slow-motion effect on the first video segment in the first video, you can slow down the second video segment in the second video that corresponds to the first video segment in time, and replace the first video segment in the first video with the third video segment obtained by the slow-motion processing to generate a new video, namely, the third video. Specifically, the mobile phone can slow down the second video segment without reducing the number of video frames of the second video segment, so as to extend the playback time of the video segment (i.e., achieve a slow-motion playback effect) without reducing the quality of the video image. For example, without reducing the number of video frames of the second video segment, reducing the frame rate of the second video segment from the second frame rate to the first frame rate is to slow down the second video segment and extend the playback time of the second video segment.

[0137] In some embodiments, the first video clip may be any video clip in the first video, or may be a video clip in the first video that includes a dynamic scene, for example, may be a video clip that includes a specified action. The second video clip corresponds to the first video clip in time, which means that the second video clip and the first video clip correspond to each other on a timeline or time axis, that is, the two correspond to the same time period and are used to present the same scene in a real environment, except that the range of content presented for the same scene may be different due to differences in the field of view of the camera. In combination with the above example, assuming that from the 5th to the 7th second, player 1 performs a shooting action a, then the first video clip from the 5th to the 7th second in the first video and the second video clip from the 5th to the 7th second in the second video both present "player 1 performs a shooting action a".

[0138] As described above, if different cameras have different field of view angles, then in videos with different frame rates, the image content or image area range presented for the same scene will be different. In some embodiments, the field of view angle of the first camera is greater than the field of view angle of the second camera, or in other words, the focal length of the first camera is shorter than the focal length of the second camera.

[0139] For a given sensor size, the size of the field of view angle is negatively correlated with the focal length. That is, the shorter the focal length, the larger the field of view angle, and conversely, the longer the focal length, the smaller the field of view angle. However, the focal length refers to the distance from the center of the lens to the focus of light gathering, which determines the degree of magnification of distant objects by the optical system. When the shooting distance remains unchanged, the shorter the focal length, the smaller the image of the object presented on the imaging plane will be, and the longer the focal length, the larger the image of the object presented on the imaging plane will be. Therefore, since the field of view angle of the first camera is greater than the field of view angle of the second camera, and the focal length of the first camera is shorter than the focal length of the second camera, the screen area of ​​the same object in the second video clip is larger than the screen area in the first video clip.

[0140] The photographed object refers to the object or element photographed by the camera in the scene. It should be noted that, since the field of view of the first camera is greater than the field of view of the second camera, when shooting the same scene, the object photographed by the second camera is generally larger than the object photographed by the first camera. The third video clip is obtained by slow processing of the corresponding video clip (i.e., the second video clip) in the second video shot by the second camera, so the screen area of ​​the same photographed object in the third video clip is also larger than its screen area in the first video clip. Therefore, after the third video clip is replaced with the first video clip in the first video to obtain the third video, the third video can present the effect of enlarging its screen area when it is played to the photographed object. In addition, the third video clip belongs to a clip composed of upscaling frames. Therefore, through the method in the embodiment of the present application, the effect of automatic upscaling and enlarging of the picture is achieved without performing a zoom operation (Zoom), and the image quality is higher (for example, higher clarity) compared to the image zooming process.

[0141] For ease of understanding, we now combine FIG. 4A to FIG. 4B See Figure 4A , 402 is the picture taken by the first camera, see Figure 4B , 404 is the picture taken by the second camera. Since the field of view of the first camera is larger than that of the second camera, the picture 402 taken by the first camera is more global and has a wider field of view. The proportion of the person (i.e., the scene object) in the picture is smaller, while in the picture 404 taken by the second camera, the proportion of the person in the picture is relatively larger. The picture 404 in the third video clip is also taken by the second camera, that is, the proportion of the person in the picture is larger. Therefore, when the picture 404 is played in the third video, the effect of upgrading and automatically zooming in can be achieved.

[0142] In some embodiments, the first camera may be a wide-angle camera, i.e., a Wide camera, and the second camera may be a telephoto camera, i.e., a Tele camera.

[0143] In some other embodiments, the first camera may be an ultra-wide-angle camera, i.e., an Ultra wide camera. The second camera may be a wide-angle camera, i.e., a Wide camera. It is understandable that an ultra-wide-angle camera has a wider field of view than a wide-angle camera, so by aligning and stitching video clips, the effect of automatically upgrading and zooming in the picture can be achieved without performing a zoom operation.

[0144] It should be noted that the first camera and the second camera are not limited to the above two combinations, and can also be other camera combinations. If you want to achieve the effect of automatic upscaling, it only needs the first camera and the second camera to be able to shoot the same scene at the same time, with different frame rates. If you want to achieve the effect of automatic upscaling + automatic zoom, it only needs the first camera and the second camera to be able to shoot the same scene at the same time, with different frame rates, and with a certain difference in field of view.

[0145] It can be understood that the first camera and the second camera also need to be relatively close in distance, that is, the relative positions of the two and the captured scene should be close, so as to ensure that the observation angles of the pictures are similar and the pictures are smoother.

[0146] It should be noted that in the embodiment of the present application, the field of view angles of at least two cameras may also be the same, that is, in videos of different frame rates shot using the method in the embodiment of the present application, the picture content or picture area range presented for the same scene may be the same or similar, and upscaling videos can also be generated.

[0147] In some embodiments, the mobile phone can control more than two cameras with different field of view angles, and respectively use different frame rates to shoot upscaled videos at the same time. Then, the size of the pictures shot by any two cameras of the more than two cameras with different field of view angles will be different. The frame rate and the size of the field of view angle can be negatively correlated, that is, the higher the frame rate of the camera, the smaller its field of view angle, and the larger the proportion of the picture shot for the same scene object. Conversely, the lower the frame rate of the camera, the larger its field of view angle, and the smaller the proportion of the picture shot for the same scene object.

[0148] Based on more than two cameras with different field of view angles shooting videos with different frame rates, an advanced upscaling and zooming video can be generated. That is, the final generated video can first upscaling and zooming in a part of the same scene, and then perform advanced zooming on the zoomed part.

[0149] for example, Figure 4C In the example, when playing the third video, the normal screen 402 may be first switched to the up-scaled zoom screen 404, and then switched to the advanced up-scaled zoom screen 406, thereby achieving an automatic multi-step up-scaled zoom effect.

[0150] The following is an example of using three cameras with different field of view angles to shoot advanced up-scaled and enlarged videos.

[0151] Assume that the three cameras with different field of view angles are the first camera, the second camera and the third camera, and the field of view angle decreases and the frame rate increases from the first camera to the third camera. Then, highlight action recognition can be performed on the video frames captured by the first camera with the lowest frame rate, and a highlight action tag can be added to the timestamp corresponding to the video frame with highlight action. The first time period and the second time period are determined based on the timestamp corresponding to the highlight action tag. The first time period can be a time period within the positive and negative preset time range of the timestamp corresponding to the highlight action tag. The first time period is adjacent to the second time period and is earlier than the second time period.

[0152] The mobile phone can determine the second video segment in the first time period from the second video captured by the second camera and process it in slow motion, and replace the first video segment in the first video in the first time period with the third video segment obtained by the slow motion. In addition, the fourth video segment in the first time period can be determined from the fourth video captured by the third camera and processed in slow motion, and the video segment in the second time period in the first video can be replaced with the fifth video segment obtained by the slow motion. Then, the unreplaced video frames in the first video are spliced ​​with the replaced third and fifth video segments to form the final advanced up-scaled video.

[0153] In some embodiments, a special recording mode is set in the mobile phone, that is, a new mode or a new path specially developed for automatically shooting high-quality upscaling videos or upscaling zoomed videos with multiple cameras, hereinafter referred to as the target recording mode. Only after entering the target recording mode can multiple cameras be controlled to automatically shoot at different frame rates at the same time to generate the final upscaling video or upscaling zoomed video.

[0154] Specifically, the mobile phone responds to the user's activation of the target video recording mode and executes the video processing method mentioned in each embodiment of the present application in the target video recording mode to automatically generate a video with an upscaling effect and high image quality of the upscaling frame. That is, the target video recording mode is a specific video recording mode for recording automatically upscaling videos. As described above, the method in the embodiment of the present application can also automatically generate upscaling and automatically zooming in videos, so the target video recording mode can also be a specific video recording mode for recording automatically upscaling and zooming in videos.

[0155] It is understandable that after entering the target video recording mode, the mobile phone can respond to the triggering operation of the shooting start control in the shooting interface to control the first camera and the second camera to start shooting video. After entering the target video recording mode, the mobile phone can also directly control the first camera and the second camera to start shooting video. The embodiments of this application do not limit the timing of starting to shoot video in the target video recording mode, as long as the first camera and the second camera can be controlled to shoot video at the same time in the target video recording mode.

[0156] In a possible implementation, in the target video recording mode, the shooting interface of the mobile phone can only display the picture collected in real time by the first camera, but not the picture collected in real time by the second camera. However, the field of view of the second camera will be prompted in the shooting interface for the user to refer to when shooting, and adjust the shooting angle so that the picture taken by the second camera is more accurate, for example, the subject in the picture taken by the second camera is more complete. It should be noted that the premise of the above scheme may be that the field of view of the first camera is greater than the field of view of the second camera. If the field of view of the first camera is smaller than the field of view of the second camera, then the picture collected in real time by the second camera can also be displayed in the shooting interface, and the field of view of the first camera can be prompted. That is, the shooting interface displays the picture collected in real time by the camera with a larger field of view, and the field of view of the camera with a smaller field of view is prompted.

[0157] In some embodiments, field of view marking information may be displayed in the shooting interface to indicate the field of view of the second camera. Field of view marking information is information used to indicate the field of view of the second camera. Exemplarily, the field of view marking information may include at least one of text prompts, marking box prompts, etc. For example, text prompt information may be displayed in the shooting interface to prompt the user how to adjust the shooting angle so that the picture captured within the field of view of the second camera is more reasonable. For another example, the field of view marking box may be directly displayed in the shooting interface, so that the field of view of the second camera is directly marked by the field of view marking box to serve as a prompt. It should be noted that in other embodiments, the field of view of the second camera may also be prompted by voice.

[0158] For ease of understanding, the following Figure 5 Provide a schematic illustration.

[0159] The mobile phone can display a shooting interface, which includes a picture preview area 501. The size of the picture preview area 501 represents the size of the field of view of the first camera. The picture preview area 501 displays the picture captured by the first camera in real time. The field of view of the first camera is greater than that of the second camera. The picture preview area 501 includes a field of view marking frame 502, which is used to mark the field of view of the second camera to guide the user to adjust the shooting angle.

[0160] like Figure 5 As shown, since the user can only see the real-time image captured by the first camera on the mobile phone, if the field of view mark box 502 is not displayed, then after starting the shooting, the head image captured by the second camera is fragmented and unreasonable. For details, see Figure 5 If the vision mark frame 502 is displayed in the screen preview area 501 according to the solution in the embodiment of the present application, the user can easily adjust the shooting angle, and in the subsequent video shooting process, the image captured by the second camera will not have the problem of head image fragmentation. For details, please refer to Figure 5 (b) in .

[0161] For example, the field of view mark box 502 can be always visible during the entire video recording process. The field of view mark box 502 can also be displayed in the shooting interface before starting the shooting, so that the user can initially understand the field of view of the second camera, and it does not need to be always visible during the video recording stage after starting the shooting. For example, during the video recording stage, the mobile phone can display the field of view mark box 502 after responding to the user's preset operation.

[0162] In another possible implementation, the shooting interface of the mobile phone can display both the images captured by the first camera in real time and the images captured by the second camera in real time. For example, the shooting interface of the mobile phone can present two image preview areas (i.e., partitioned display), the first image preview area is used to display the images captured by the first camera in real time, and the second image preview area is used to display the images captured by the second camera in real time. Thus, the user can understand the field of view of the first camera and the field of view of the second camera through the shooting interface, achieve the purpose of accurate shooting, and facilitate subsequent video processing.

[0163] As mentioned above, the mobile phone needs to enter the target recording mode. In the target recording mode, prompt information for indicating the field of view of the second camera is displayed in the shooting interface. Therefore, the specific implementation method of the mobile phone entering the target recording mode will be explained as an example.

[0164] In a possible implementation, the mobile phone can enter the target video recording mode based on the shooting interface. Next, how to enter the target video recording mode based on the shooting interface will be described.

[0165] Specifically, the mobile phone can display the entrance to the target recording mode in the shooting interface. The mobile phone can respond to the trigger operation (i.e., video shooting operation) of the entrance to the target recording mode in the shooting interface to start or enter the target recording mode, and control the first camera and the second camera to shoot high-speed video in this target recording mode.

[0166] In one example, the entry of the target video mode can be associated with the existing video mode. Specifically, the entry of the target video mode can be added to the shooting interface of the existing video mode. Figure 6 As shown, the "Movie" mode is an existing video recording mode. An option of the "Upscale" video recording mode (i.e., the target video recording mode) is added to the shooting interface of the "Movie" mode. By responding to the user's triggering operation on the option, the "Upscale" video recording mode can be entered, and the field of view mark box 502 is displayed in the picture preview area 501 of the shooting interface.

[0167] In another example, the target recording mode does not need to be started by relying on the existing recording mode. The target recording mode can be used as an independent recording mode, and the entrance to the target recording mode can be set in the mode selection area of ​​the shooting interface. The mobile phone can enter the target recording mode in response to the triggering operation of the entrance to the target recording mode in the mode selection area. After entering the target recording mode, a field of view mark box is displayed in the shooting interface to mark the field of view of the second camera.

[0168] In another possible implementation, the mobile phone may also start or enter the target video recording mode without the aid of the shooting interface. Specifically, the mobile phone may start or enter the target video recording mode in response to a preset designated operation (i.e., a video shooting operation). The preset designated operation may include a preset combination operation or a preset single specific operation. For example, the user presses button 1+button 2+button 3 at the same time to input the video shooting operation and start the target video recording mode.

[0169] In other embodiments, the mobile phone may also be provided with a dedicated button or paddle for the target video recording mode at the hardware level, and the mobile phone may start or enter the target video recording mode in response to the user pressing the button or paddle. The embodiments of the present application do not limit the specific method of starting the target video recording mode.

[0170] It can be understood that in order to ensure the image quality of the captured video frames, the mobile phone needs to perform 3A control on the first camera and the second camera during the process of controlling the first camera and the second camera to shoot video simultaneously in the target recording mode.

[0171] For ease of understanding, the principles of the 3A technology and how to perform 3A control in the embodiments of the present application will be explained below.

[0172] 3A technology uses 3A algorithms to achieve 3A control of the camera to help the camera automatically adjust parameters to obtain better image quality. 3A algorithms include auto focus (AF) algorithm, auto exposure (AE) algorithm and auto white balance (AWB) algorithm, which are three algorithms that automatically adjust camera parameters. 3A control includes auto focus control, auto exposure control and auto white balance control.

[0173] Among them, autofocus (AF) control refers to automatically adjusting the distance between the camera lens and the photosensitive chip (ie, sensor) through an autofocus algorithm to ensure that the subject (the object being photographed) presents a clear image on the imaging plane.

[0174] Automatic exposure (AE) control refers to the continuous detection of the exposure status of the captured images (or video frames). If the exposure status of the previously captured images is found to be abnormal, the automatic exposure algorithm can automatically adjust the corresponding camera parameters (for example, automatically adjust the shutter speed, aperture size, ISO and other camera parameters) so that the exposure status of the subsequently captured video frames is closer to the target exposure result, thereby restoring the exposure status to normal as soon as possible. This process is continuous, that is, during the video shooting process, the automatic exposure algorithm will continuously monitor the changes in scene light, and analyze and adjust in time to keep the exposure status of the video frame always appropriate.

[0175] Automatic white balance (AWB) control continuously detects the color temperature of scene light and automatically adjusts the camera's white balance parameters (for example, the gain values ​​of the three primary colors of red, green and blue, color temperature offset, and other parameters) based on the detection results through the automatic white balance algorithm, thereby eliminating color casts in the captured image and making the white in the image look more accurate and natural.

[0176] It should be understood that video shooting is a process of continuously shooting video frames, so the above-mentioned 3A controls are continuous and persistent, that is, before shooting each video frame, the mobile phone will calculate the 3A control information for the first camera and the second camera respectively based on the 3A algorithm, and when shooting the video frame, the first camera and the second camera will be respectively controlled by 3A according to the 3A control information corresponding to each camera.

[0177] 3A control information is the information needed to perform 3A control on the camera. 3A control information can be a control instruction or adjustment instruction for the camera in 3A. For example, for exposure control, if the algorithm module finds that the image is overexposed after the current frame is taken, it will send an instruction to the sensor, instructing it to adjust the ISO in the next frame to avoid overexposure. Then, sending an instruction to adjust the ISO to the sensor is a 3A control information. For another example, for color correction, indicating how much B / G (blue-green ratio) needs to be supplemented when shooting the next frame is a 3A control information.

[0178] As mentioned above, 3A control needs to be implemented based on 3A control information. Next, how to obtain the 3A control information will be introduced.

[0179] In some embodiments, for any one of the first camera and the second camera of the embodiment of the present application, during the process of the camera shooting a video, the mobile phone can continuously perform statistical analysis on the previously shot video frames (or image frames), so as to predict the 3A control information corresponding to the camera during subsequent shooting based on the 3A algorithm combined with the statistical analysis results, that is, to determine how to control and adjust the lens focal length, exposure, and white balance when the camera shoots video frames subsequently. Furthermore, during subsequent shooting, the sensor's lens focal length adjustment (i.e., focus adjustment), exposure state adjustment, and white balance adjustment can be automatically implemented based on the 3A control information to shoot a video with higher image quality.

[0180] Specifically, before shooting each video frame, the mobile phone can perform statistical analysis on the previously shot video frames to predict the 3A control information used for the camera when shooting the video frame this time based on the 3A algorithm combined with the statistical analysis results.

[0181] The previously shot video frame is a video frame that has been shot before the currently shot video frame and serves as a reference for the 3A control of the currently shot video frame. For example, before the second video frame is shot, the first video frame belongs to the previously shot video frame. The embodiment of the present application does not make too many restrictions on the scope of the previously shot video frame. As long as it serves as a reference for the 3A control of the subsequent shooting, it can belong to the previously shot video frame, and is not limited to the previous video frame.

[0182] Exemplarily, the statistical analysis of the previously shot video frames may include a statistical analysis of the image parameters of the previously shot video frames. Specifically, for any one of the first camera and the second camera of the embodiment of the present application, the mobile phone can continuously perform statistical analysis on the image parameters of the video frames previously shot by the camera to obtain image statistical parameters. It can be understood that a video frame is an image frame, which is essentially an image, so it has corresponding image parameters. The mobile phone can calculate based on the image statistical parameters according to the 3A control algorithm to obtain the corresponding 3A control information when the camera subsequently shoots video frames.

[0183] After obtaining the 3A control information corresponding to the first camera and the second camera, any one of the first 3A control mode and the second 3A control mode can be adopted to perform 3A control on the first camera and the second camera. The above two 3A control modes will be explained below.

[0184] The first 3A control mode: that is, 3A synchronous control.

[0185] If the first 3A control method is used, then during the process of the second camera shooting a video, the mobile phone will refer to the first 3A control information corresponding to the first camera to perform 3A control on the second camera to obtain the second video. Specifically, when the mobile phone uses the 3A algorithm to calculate the 3A control information corresponding to the second camera, it will refer to the first 3A control information corresponding to the first camera.

[0186] In a possible implementation, for the first 3A control mode, the mobile phone can refer to the first 3A control information, and in combination with the second 3A control information corresponding to the second camera itself, determine the third 3A control information that is ultimately used to perform 3A control on the second camera. In the process of shooting a video with the second camera, the mobile phone can perform 3A control based on the third 3A control information to obtain the shot second video. Among them, the third 3A control information is closer to the first 3A control information than the second 3A control information. It can be understood that since the third 3A control information is closer to the first 3A control information than the second 3A control information, the image frames captured by the second camera and the first camera can be closer in terms of exposure, color, etc. Since the image frames captured by the two cameras need to be spliced ​​to generate a scaled-up video (i.e., the third video) later, the 3A synchronous control can make the screen switching transition more natural when playing the scaled-up segment in the scaled-up video.

[0187] It can be understood that the 3A control information corresponding to each camera itself is determined based on the image statistical parameters of the video frame collected by the camera. Therefore, the first 3A control information is calculated based on the first image statistical parameters. The second 3A control information is calculated based on the second image statistical parameters.

[0188] The first image statistical parameter is obtained by counting the image parameters of the video frame collected by the first camera. The second image statistical parameter is obtained by counting the image parameters of the video frame collected by the second camera. Specifically, in the process of controlling the first camera and the second camera to shoot video in real time, the mobile phone can count the image parameters of the video frame collected by the first camera to obtain the first image statistical parameter. Similarly, the mobile phone can count the image parameters of the video frame collected by the second camera to obtain the second image statistical parameter.

[0189] In some embodiments, the mobile phone has a first 3A module and a second 3A module. The mobile phone can use the first 3A module to calculate the first 3A control information according to the first image statistical parameters using a 3A algorithm. The mobile phone can use the second 3A module to calculate the second 3A control information according to the second image statistical parameters using a 3A algorithm.

[0190] When the first 3A control mode (i.e., 3A synchronous control) is adopted, the mobile phone can synchronize the first 3A control information to the second 3A module through the first 3A module, and the second 3A module can refer to the synchronized first 3A control information and determine the third 3A control information in combination with the second 3A control information (i.e., the second 3A control information calculated by the second 3A module according to the second image statistical parameters). The third 3A control information is used to finally perform 3A control on the second camera.

[0191] In some embodiments, the mobile phone is provided with a first ISP online front end and a second ISP online front end; the first ISP online front end corresponds to the first camera and is used to process the video frames captured by the first camera; the second ISP online front end corresponds to the second camera and is used to process the video frames captured by the second camera.

[0192] Therefore, the mobile phone can count the image parameters of the video frames collected by the first camera through the first ISP online front end to obtain the first image statistical parameters. The mobile phone can transmit the first image statistical parameters to the first 3A module through the first ISP online front end, so that the first 3A module calculates the first 3A control information based on the first image statistical parameters.

[0193] The mobile phone can count the image parameters of the video frames collected by the second camera through the second ISP online front end to obtain the second image statistical parameters. The mobile phone can transmit the second image statistical parameters to the second 3A module through the second ISP online front end, so that the second 3A module calculates the second 3A control information based on the second image statistical parameters.

[0194] Exemplarily, the first 3A module and the second 3A module may be provided in the ISP.

[0195] Exemplarily, the mobile phone may average the first 3A control information and the second 3A control information to obtain the third 3A control information. The mobile phone may also perform weighted calculation on the first 3A control information and the second 3A control information according to preset weights to obtain the third 3A control information. There is no limitation to this, as long as the first 3A control information can be referred to, and the second 3A control information is not used alone to perform 3A control on the second camera.

[0196] For ease of understanding, an example is given below. Taking color correction control as an example, the first 3A control information may include a first adjustment ratio of B / G (i.e., blue / green), for example, B / G=3, and the second 3A control information may include a second adjustment ratio of B / G, for example, B / G=2. Then, the third 3A control information may include B / G=(3+2) / 2=2.5. Therefore, the ratio of B / G=2.5 may be used to control the shooting of the next video frame.

[0197] In another possible implementation, referring to the first 3A control information, the second camera may be 3A controlled only according to the first 3A control information, without combining with the second 3A control information corresponding to the second camera. It can be understood that the 3A control only according to the first 3A control information includes the following two methods:

[0198] Method 1: The mobile phone can directly use the first 3A control information as the third 3A control information ultimately used to perform 3A control on the second camera.

[0199] Exemplarily, the mobile phone can analyze the difference in image content within the respective fields of view of the first camera and the second camera, and when the difference in image content meets the preset conditions, the first 3A control information can be used as the third 3A control information for finally performing 3A control on the second camera. If the difference in image content is large, for example, the field of view of the second camera is pure black, and the field of view of the first camera is white and black, and the black is only a part of the image content in the central area, then it is not appropriate to use the first 3A control information as the third 3A control information for finally performing 3A control on the second camera.

[0200] Method 2: The mobile phone can also make adjustments based on the first 3A control information based on the 3A control characteristics of the second camera to obtain the third 3A control information.

[0201] It can be understood that the first camera and the second camera have their own 3A control characteristics respectively. The 3A control characteristics of the two cameras can reflect the difference in the bias of the two cameras in 3A control to a certain extent. Therefore, based on the first 3A control information and the 3A control characteristics of the second camera, the second camera can be more accurately 3A controlled. For example, the field of view of the Tele camera is smaller than that of the Wide camera. Therefore, in general, when the Tele camera is complementing colors, the degree of color complementation will be greater than that of the Wide camera. This is the color complement characteristic of the Tele camera. Then, the color complement ratio of the Tele camera can be determined by referring to the color complement ratio of the Wide camera (i.e., the first 3A control information) and combining the color complement characteristics of the Tele camera, so that the 3A control of the Tele camera is closer to the 3A control of the Wide camera.

[0202] The second 3A control mode: that is, 3A independent control.

[0203] If the second 3A control method is adopted, then during the video shooting process, the mobile phone will independently perform 3A control on the first camera and the second camera, that is, each camera (that is, any one of the first camera and the second camera) will not refer to the other party's 3A control information, but only use the video frame taken by itself first to calculate the 3A control information corresponding to the camera.

[0204] Specifically, when the first camera is shooting a video, the mobile phone can perform 3A control on the first camera based on the first 3A control information to shoot the first video. When the second camera is shooting a video, the mobile phone can perform 3A control on the second camera based on the second 3A control information to shoot the second video.

[0205] It can be understood that the first 3A control mode and the second 3A control mode can be determined according to the settings, and an entry can also be provided in the interface for the user to select the required 3A control mode, and there is no limitation on this.

[0206] After shooting the first video and the second video according to the methods mentioned in the above embodiments, the mobile phone can determine the video segment in the first video that needs to achieve the upscaling effect, that is, the first video segment. The mobile phone can determine the second video segment that corresponds to the first video segment in time from the second video, and slow down the second video segment to obtain a third video segment. The mobile phone can replace the first video segment in the first video with the third video segment to obtain a third video.

[0207] It can be understood that the second video clip and the first video clip correspond to the same time period on the timeline.

[0208] In a possible implementation, the first video segment is a highlight segment in the first video; the highlight segment corresponds to a target time period on the timeline of the first video; and the second video segment is a video segment in the second video that is within the target time period.

[0209] Highlight clips, also known as highlight clips, are video clips consisting of a single video frame or multiple consecutive video frames that record the wonderful moments in the video. Wonderful moments can also be called wonderful moments, which can include moments when highlight actions such as laughing, running, jumping, throwing, shooting, hitting the ball, looking back, etc. occur. Therefore, highlight clips can be video clips corresponding to highlight actions.

[0210] In some embodiments, the highlight segment can be automatically identified. The following are the steps for the mobile phone to automatically identify the highlight segment:

[0211] Specifically, the mobile phone can perform highlight action recognition on the video frames used to form the first video to identify video frames with highlight actions (i.e., video frames that record wonderful moments or highlight moments). Furthermore, the mobile phone can determine highlight segments from the first video based on the video frames with highlight actions.

[0212] The highlight clip includes a continuous video frame sequence (i.e., a portion of the video frame sequence in the entire first video), and the video frame sequence includes the video frame with the highlight action. It can be understood that the highlight clip is used to record the highlight action that occurs, so only a portion of the entire first video is used to record the highlight action that occurs.

[0213] Exemplarily, the highlight segment includes the video frame with highlight motion, and adjacent preceding and / or adjacent succeeding partial video frames. The adjacent preceding and / or adjacent succeeding partial video frames refer to video frames that are before and / or after the video frame with highlight motion and are adjacent to the video frame with highlight motion. Adjacent may include directly adjacent or indirectly adjacent. That is, the highlight segment includes a continuous sequence of video frames, and the video frame sequence includes the video frame with highlight motion.

[0214] In some embodiments, the target time period corresponding to the highlight segment is determined based on the timestamp of the video frame with the highlight action. Specifically, after the mobile phone identifies the video frame with the highlight action, it can expand the preset time range based on the timestamp of the video frame with the highlight action to obtain the target time period. For example, based on the timestamp of the video frame with the highlight action, expand by plus or minus 1 second to obtain the target time period or target time interval.

[0215] The video segment in the first video that is within the target time period is the highlight segment, that is, the first video segment. The mobile phone can determine the video segment that is within the target time period from the second video to obtain the second video segment.

[0216] Exemplarily, the first video has metadata carrying a highlight action tag. The highlight action tag in the metadata has a corresponding timestamp on the time axis of the first video; and the target time period is a time period determined according to the timestamp.

[0217] Specifically, the mobile phone can add a highlight action tag corresponding to the timestamp of the video frame in which the highlight action is identified, and obtain metadata of the first video carrying the highlight action tag. The mobile phone can determine the target time period based on the timestamp corresponding to the highlight action tag in the metadata. It can be understood that the timestamp corresponding to the highlight action tag is the timestamp of the video frame with the highlight action.

[0218] For ease of understanding, now combined Figure 7 The first video clip and the second video clip are schematically described. The original video shot by the Wide camera (hereinafter referred to as the "Wide original video" for brevity) is the first video, and the original video shot by the Tele camera (hereinafter referred to as the "Tele original video") is the second video. Figure 7 It can be seen intuitively that the time length of the Wide original video and the Tele original video is the same. According to the timestamp corresponding to the wonderful tag (i.e., wonderful TAG, also called highlight action tag) in the Wide original video, the time period of plus or minus 1s of the timestamp can be determined as the target time period, and the video segment in the Wide original video that is within the target time period is the first video segment. Figure 7 There are multiple wonderful tags in , then each wonderful tag will correspond to a first video clip, for example, Figure 7 701a and 701b in the figure belong to the first video segments corresponding to different wonderful tags. According to the target time segment corresponding to the first video segment, a video segment in the target time segment can be intercepted from the Tele original video as the second video segment. Figure 7 702a in the Tele original video is the second video segment corresponding to the same target time period as the first video segment 701a, and 702b in the Tele original video is the second video segment corresponding to the same target time period as the first video segment 701b.

[0219] In some embodiments, the highlight action may be an action specified by the user, that is, the highlight action may be pre-specified, and subsequently, the video frame with the pre-specified highlight action may be automatically identified to obtain the highlight segment.

[0220] Exemplarily, before controlling the first camera and the second camera to shoot, the mobile phone can determine the designated and identified highlight action in response to the user's action designation operation. Specifically, the mobile phone can provide an action designation interface or an action designation portal and present it to the user. Obtain the designated information or custom information entered by the user in the action designation interface or the action designation portal. When the mobile phone controls the first camera and the second camera to shoot a video at the same time, or stops shooting a video, the highlight segment that matches the custom information is identified from the first video shot by the first camera, which is the first video segment.

[0221] In another possible implementation, the first video segment may also be any video segment specified by the user. For example, the user may specify which video content the user wants to perform upscaling processing on, and the mobile phone may determine the segment specified by the user as the first video segment.

[0222] Specifically, after controlling the first camera and the second camera to shoot a video, the mobile phone can determine the designated first video segment in response to the user's segment designation operation. Optionally, after controlling the first camera and the second camera to shoot the first video and the second video, the mobile phone can present a segment designation interface, through which the user's segment designation operation for the first video is received to determine the first video segment designated by the segment designation operation. It can be understood that in an embodiment, the segment designation interface can present candidate video segments for the user to select and designate, and the mobile phone can determine the selected candidate video segment as the first video segment in response to the user's selection operation on the candidate video segment, or the mobile phone can determine the candidate video segment remaining after the deletion as the first video segment in response to the user's deletion operation on the candidate video segment.

[0223] In another possible implementation, the first video clip may be a video clip within a preset time period. For example, the upscaling effect may be specified at the head or end of the video. In this case, the first video clip may be a video clip within a preset time period at the head or end of the video.

[0224] It can be understood that after determining the second video segment, the mobile phone can slow down the second video segment to obtain a third video segment. Specifically, the mobile phone can adjust the inter-frame interval between the video frames in the second video segment so that the frame rate of the second video segment is reduced from the second frame rate to the first frame rate, thereby achieving slow processing of the second video segment. Furthermore, the mobile phone can discard the first video segment in the first video, and replace the first video segment with the third video segment (i.e., the second video segment after slow processing), and splice it with the remaining video frames in the first video to obtain the third video.

[0225] for example, Figure 8In the example, the second video segments 702a and 702b are intercepted from the Tele original video. Since the frame rate of the Tele original video is 60FPS, the number of video frames of the intercepted second video segment is twice the number of video frames of the first video segment in the Wide original video. Furthermore, the mobile phone can adjust the timestamps of the video frames in the second video segment to increase the inter-frame interval, so that the frame rate of the second video segment is reduced from the second frame rate 60FPS to the first frame rate 30FPS, that is, the frame rate of the third video segment obtained after adjustment is 30FPS. Then, in the case where the number of video frames is twice that of the first video segment, if the frame rate is adjusted to be the same as that of the first video segment, 2X slow speed (i.e., 2 times slow speed) can be achieved, that is, the playback time of the third video segment 703a is extended by 2 times relative to the second video segment 702a, and the playback time of the third video segment 703b is extended by 2 times relative to the second video segment 702b. Then, the third video segment 703a replaces 701a in the Wide original video, and the third video segment 703b replaces 701b in the Wide original video, and the video frames are spliced ​​to obtain an upscaled video. Since the duration (playing duration) of the third video segment is extended, Figure 8 The length of the medium-speed video is also longer than the wide original video.

[0226] In some embodiments, Fig. 9 As shown, a flow chart of a video processing method is provided. The method may specifically include the following steps:

[0227] S901, in a target recording mode, simultaneously controlling a first camera and a second camera to shoot videos using different frame rates to obtain a first video and a second video, wherein the first video corresponds to a first frame rate, the second video corresponds to a second frame rate, and the first frame rate is less than the second frame rate.

[0228] It should be understood that the method of entering or starting the target recording mode can be adopted Figure 6 For example, the shooting interface includes an entry to the target video mode. In response to a trigger operation on the entry to the target video mode, the target video mode is started or entered, and the field of view indication information is displayed in the picture preview area of ​​the shooting interface.

[0229] It should be noted that the above also mentions other ways to enter the target recording mode, and is not limited to Figure 6 The method shown.

[0230] In the process of controlling the first camera and the second camera to shoot a video, the mobile phone can use the above-mentioned first 3A control method or the second 3A control method to perform 3A control on the first camera and the second camera.

[0231] To facilitate understanding of the process of shooting the first video and the second video, Fig.10 A more detailed explanation is given. Fig.10 A schematic diagram of the principle of a video shooting process provided in an embodiment of the present application.

[0232] See also Fig.10 , the first camera is a Wide camera (corresponding to a first frame rate of 30FPS), and the second camera is a Tele camera (corresponding to a second frame rate of 60FPS). The electronic device also includes an ISP, wherein the ISP includes an ISP front end and an ISP back end, and the ISP front end includes two ISP online front ends (i.e., ISP online front end 1 and ISP online front end 2) and one ISP offline front end.

[0233] like Fig.10 As shown, ISP online front end 1 is responsible for processing the data collected by the Wide camera, and ISP online front end 2 is responsible for processing the data stream collected by the Tele camera. For example, ISP online front end 1 and ISP online front end 2 can perform processing such as removing bad pixels and collecting statistical information.

[0234] The statistical information collection may include the collection of image statistical parameters. It should be understood that during the video shooting process, it is necessary to continuously perform 3A control on the Wide camera and the Tele camera respectively to ensure the quality of the shot video. Therefore, each ISP online front end will perform image parameter statistical analysis on the image stream collected by the camera it is responsible for, that is, ISP online front end 1 will perform image parameter statistics on the image stream collected by the Wide camera to obtain the first image statistical parameter, and ISP online front end 2 will perform image parameter statistics on the image stream collected by the Tele camera to obtain the second image statistical parameter.

[0235] These image statistical parameters can be used as basic data for the subsequent generation of 3A control information. Specifically, the ISP online front end 1 transmits the first image statistical parameters to the first 3A module, and the first 3A module calls the 3A algorithm in the hardware abstraction layer to calculate the first 3A control information for the Wide camera based on the first image statistical parameters. The ISP online front end 2 transmits the second image statistical parameters to the second 3A module, and the second 3A module calls the 3A algorithm in the hardware abstraction layer to calculate the second 3A control information for the Wide camera based on the second image statistical parameters. The hardware abstraction layer returns the calculated first 3A control information to the first 3A module, and returns the second 3A control information to the second 3A module.

[0236] It should be noted that, since the video clips captured by the two cameras will be stitched together later, in order to make the picture quality of the stitched video more harmonious and unified, the 3A control situation or 3A control decision of the Wide camera will be referred to, and the Tele camera will be controlled by 3A. Therefore, 3A synchronous control will be involved.

[0237] Specifically, the first 3A module will synchronize the first 3A control information to the second 3A module. The second 3A module can call an algorithm to refer to the first 3A control information, and combine the second 3A control information calculated based on the image collected by the Tele camera itself to determine the third 3A control information that is ultimately used to perform 3A control on the Tele camera.

[0238] It can be understood that in subsequent video shooting, the mobile phone can perform 3A control on the Wide camera according to the first 3A control information, and perform 3A control on the Tele camera according to the third 3A control information, so that the picture quality captured by the two cameras is closer.

[0239] The two ISP online front ends can input the processed data of the two cameras to the ISP offline front end, and the ISP offline front end processes the data of the two cameras using a time-division multiplexing technology.

[0240] Time Division Multiplexing (TDM) refers to dividing the time of a physical line into multiple small time periods, each of which is used to transmit a separate signal or data stream. In this way, multiple signals can be transmitted simultaneously on the same physical line, thereby achieving the purpose of multiplexing. In conjunction with the solution of the present application, the time of using the ISP offline front end can be divided into multiple small time periods, each of which is used to process the data of one of the cameras, and the data of the two cameras are processed alternately, so that the data of the two cameras can be quickly processed by the ISP offline front end.

[0241] For example, the ISP offline front end can process the data of the Wide camera at the 1st ms, the data of the Tele camera at the 2nd ms, the data of the Wide camera again at the 3rd ms, and the data of the Tele camera at the 4th ms, and so on. Exemplarily, in order to ensure the timeliness of data processing, the data cached between the two time periods can also be processed in advance. For example, if the data of the Tele camera is processed at the 2nd ms, some data can be processed in advance before the 2nd ms arrives, so that the processing time efficiency is higher.

[0242] The ISP offline front end can input the processed images to the ISP back end, and the ISP back end uses the time-division multiplexing technology to process the images of the two cameras input by the ISP offline front end, and obtains images with relatively high image quality that meet the standards and can be used to form a video. The time-division multiplexing processing principle of the ISP back end and the ISP offline front end is the same, which will not be repeated here.

[0243] In this way, after continuous processing by the ISP, each image that meets the standard is obtained, and finally each image that meets the standard with a time sequence can form the first video Video1 corresponding to the Wide camera and the second video Video2 corresponding to the Tele camera. In addition, during the video shooting process, the picture collected by the Wide camera in real time can be previewed and displayed in real time in the interface of the mobile phone. For example, the picture collected by the Wide camera in real time is displayed in real time in the picture preview area of ​​the shooting interface. It can be understood that the picture collected by the Wide camera in real time can be the image picture processed by the ISP backend.

[0244] from Fig.10 It can be seen that in addition to inputting the images of the two cameras after the ISP offline front-end processing into the ISP back-end, the images of the Wide camera after the ISP offline front-end processing can also be input into the perception engine, and the perception engine calls the wonderful moment recognition algorithm (or the highlight action recognition algorithm) to perform wonderful moment recognition processing (or highlight action recognition processing) on ​​each image, and tags the timestamps of the images with wonderful moments (i.e., highlight action tags, also called wonderful tags), and saves the TAG corresponding to the timestamp as metadata Metadata to the metadata track of the first video Video1. The metadata can be used to determine the first video segment from the first video, and the second video segment from the second video. The specific processing can be illustrated in the example under step S902.

[0245] S902, determining a first video segment for which a slow motion effect needs to be achieved from the first video, and determining a second video segment corresponding to the same time period as the first video segment from the second video.

[0246] Exemplarily, the method for determining the first video segment may adopt the method of automatically identifying highlight segments mentioned above.

[0247] Optionally, metadata having a highlight action tag corresponding to the first video may be used to determine the second video segment from the second video.

[0248] S903, slow down the second video segment to obtain a third video segment; the third video segment corresponds to the first frame rate.

[0249] S904: Replace the first video segment in the first video with the third video segment to obtain a third video.

[0250] It is understandable that the frame rate of the third video is also the first frame rate, so the third video can be played at the first frame rate later. However, since the third video segment in the third video is obtained by slowing down the video segment with a high frame rate, a slow motion effect or a speed-up effect with a longer playing time can be presented when the third video segment is played.

[0251] In some embodiments, the position of the first video clip in the first video can be recorded as the target position, and the mobile phone can discard the first video clip in the first video; insert the third video clip into the target position in the first video, and splice the inserted third video clip with the remaining video frames in the first video to obtain the third video.

[0252] It can be understood that the target position can reflect the positional relationship of the first video clip in the first video compared with other video frames or video clips. After the third video clip is subsequently inserted into the target position, it can ensure that the picture or scene transition in the subsequently generated third video is smooth and does not jump.

[0253] The third video segment is obtained by slow-motion processing of the second video segment. The number of video frames in the third video segment remains unchanged, but the interval between frames becomes larger, so the time period corresponding to the third video segment on the time axis also becomes longer. Figure 8 The upscaled video is the third video. The first video segments 701a and 701b in the first video (i.e., the original video shot by the Wide camera) are discarded and replaced with the third video segment 703a obtained by slowing down the second video segment 702a and the third video segment 703b obtained by slowing down the second video segment 702b, so that the upscaled video can be obtained. Figure 8 It can be seen that the corresponding time period (ie, the playing time) of the third video clips 703a and 703b in the up-scaled video on the time axis is extended by 2 times.

[0254] In some embodiments, the native audio of the time period corresponding to the third video clip on the timeline can be hidden and replaced with a preset background music BGM to avoid the problem of native audio mismatch caused by slow processing. For example, it can be replaced with a preset soft or cheerful background music. Exemplarily, the background music that matches the type of highlight action identified by the perception engine can be determined. For example, if the highlight action is identified as practicing martial arts, then exciting background music can be added to the highlight clip. For another example, if the highlight action is identified as looking back, then soft background music can be added to the highlight clip.

[0255] In some embodiments, the mobile phone can also add a switching video transition effect at the splicing point between the third video clip and the remaining video frames in the first video, so that the transition is more natural and smooth when the automatic upscaling or automatic upscaling zoomed-in picture is presented. Specifically, for each third video clip, the third video clip has two splicing points with the first video, namely the first splicing point and the second splicing point. The first splicing point is the splicing point between the starting position of the third video clip and the first video, and the second splicing point is the splicing point between the ending position of the third video clip and the first video. Then, video transition effects can be added to both the first splicing point and the second splicing point. Video transition effects can include image effects and / or audio effects. It can be understood that video transition effects may not be added at the splicing point.

[0256] For example, in addition to the different frame rates of the first camera and the second camera (i.e., the first video corresponds to the first frame rate, the second video corresponds to the second frame rate, and the first frame rate is less than the second frame rate), the field of view of the first camera is larger than the field of view of the second camera. Then, the screen area of ​​the same scene object in the third video segment is larger than its screen area in the first video segment. Therefore, when the third video is played to the third video segment, not only a slow motion effect or a scaled-up effect can be achieved, but also an automatic zoom effect can be achieved. Figure 4B and Figure 4C shown.

[0257] In some embodiments, the mobile phone can save the third video as well as the first video and the second video. For example, the above three videos can be saved in the gallery.

[0258] In other embodiments, the mobile phone may only save the third video. For example, in order to save storage space of the mobile phone, after the third video is generated based on the first video and the second video, the first video and the second video may be deleted, and only the third video may be retained in the gallery. This is not limited. It is understood that the automatic deletion of the first video and the second video may be performed after authorization by the user.

[0259] An embodiment of the present application also provides a chip system, which is applied to an electronic device. The chip system includes one or more processors, and the processor is used to call computer instructions so that the electronic device executes each function or step in the above method embodiment.

[0260] An embodiment of the present application also provides a computer-readable storage medium, which includes computer instructions. When the computer instructions are executed on the above-mentioned electronic device, the terminal device executes each function or step in the above-mentioned method embodiment.

[0261] The embodiment of the present application also provides a computer program product. When the computer program product runs on a computer, it enables the computer to execute each function or step in the above method embodiment.

[0262] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0263] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0264] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0265] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0266] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0267] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application shall be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A video processing method, characterized in that: Applied to an electronic device, the electronic device comprises a first camera and a second camera; the lenses of the first camera and the second camera face the same direction; the method comprises: In response to a video shooting operation, simultaneously controlling the first camera to shoot a first video using a first frame rate, and controlling the second camera to shoot a second video using a second frame rate; the first frame rate is lower than the second frame rate; Save the third video; Among them, the third video is a video obtained by replacing the first video segment in the first video with the third video segment; the third video segment is obtained by slowing down the second video segment in the second video; the second video segment and the first video segment correspond to the same time period on the timeline.

2. The method according to claim 1, characterized in that The field of view of the first camera is greater than the field of view of the second camera; the picture area of ​​the same object in the third video clip is greater than the picture area in the first video clip.

3. The method according to claim 2, characterized in that The method further comprises: Displaying a shooting interface; the shooting interface includes a picture preview area, and the picture preview area includes field of view marking information; the picture preview area is used to display the picture collected in real time by the first camera using the first frame rate; the field of view marking information is used to mark the field of view of the second camera; In response to the video shooting operation, simultaneously controlling the first camera to shoot a first video using a first frame rate, and controlling the second camera to shoot a second video using a second frame rate, comprises: In response to a video shooting operation triggered in the shooting interface, the first camera is controlled to shoot a first video using a first frame rate, and the second camera is controlled to shoot a second video using a second frame rate.

4. The method according to claim 3, characterized in that The shooting interface includes an entrance to a target video recording mode; The target video recording mode is a specific video recording mode for recording automatic upscaling video; The method further comprises: In response to a triggering operation on the entrance, in the target video recording mode, the field of view marking information is displayed in a picture preview area of ​​the shooting interface.

5. The method according to claim 1, characterized in that The first video is shot when the first camera is controlled by 3A using the first 3A control information; the second video is shot when the second camera is controlled by 3A with reference to the first 3A control information corresponding to the first camera.

6. The method according to claim 5, characterized in that The first 3A control information is calculated based on a first image statistical parameter corresponding to a video frame captured by the first camera; The controlling the second camera to shoot the second video using the second frame rate includes: Controlling the second camera to shoot a video using a second frame rate, and performing 3A control based on third 3A control information during the video shooting process to obtain a shot second video; The third 3A control information is closer to the first 3A control information than the second 3A control information; the second 3A control information is calculated based on second image statistical parameters corresponding to the video frame captured by the second camera.

7. The method according to claim 6, characterized in that The electronic device comprises a first 3A module and a second 3A module; the first 3A control information is calculated by the first 3A module based on the first image statistical parameter; The second 3A control information is calculated by the second 3A module based on the second image statistical parameters; The method further comprises: Synchronize the first 3A control information calculated by the first 3A module to the second 3A module; The third 3A control information is determined by the second 3A module according to the second 3A control information and the first 3A control information synchronized thereto.

8. The method according to claim 1, characterized in that: The first video segment is a highlight segment in the first video; the highlight segment corresponds to a target time period on the timeline of the first video; and the second video segment is a video segment in the second video that is within the target time period.

9. The method according to claim 8, characterized in that The first video has metadata carrying a highlight action tag; the highlight action tag in the metadata has a corresponding timestamp; and the target time period is a time period obtained by expanding a preset time range based on the timestamp.

10. The method according to claim 9, characterized in that The method further comprises: Performing highlight action recognition processing based on the video frames captured by the first camera; A highlight action tag is added corresponding to the timestamp of the video frame in which the highlight action is identified, and metadata carrying the highlight action tag of the first video is obtained.

11. The method according to any one of claims 1 to 10, characterized in that The method further comprises: discarding the first video segment in the first video; The third video clip is inserted into a target position in the first video, and the inserted third video clip is spliced ​​with the remaining video frames in the first video to obtain the third video; the target position is the position of the first video clip in the first video.

12. An electronic device, characterized in that: The electronic device includes at least: a first camera and a second camera, a memory and one or more processors; the first camera and the second camera are used to capture images, and the memory is used to store computer instructions. When the one or more processors execute the computer instructions, the electronic device executes the method as described in any one of claims 1-11.

13. A chip system, characterized in that: The chip system is applied to an electronic device, and the chip system includes one or more processors, and the processor is used to call computer instructions so that the electronic device executes the method as described in any one of claims 1-11.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes computer instructions, and when the computer instructions are executed on an electronic device, the electronic device executes the method according to any one of claims 1 to 11.

15. A computer program product, characterized in that When the computer program product is run on a computer, the computer is caused to execute the method according to any one of claims 1 to 11.

Images