Video processing method, electronic equipment and computer readable storage medium

Abstract

The embodiment of the invention discloses a video processing method, electronic equipment and a computer readable storage medium, and the method comprises the steps: the electronic equipment obtains original image data and metadata in a video recording process, processes the original image data through an image signal processor, and obtains first video data; when video recording is quitted, a first video file is obtained, the first video file and the metadata are stored, and the first video file is a video file generated by encoding the first video data; the gallery application calls required metadata according to a video algorithm needing to be executed, and processes the first video file to obtain a second video file; the video quality of the second video file is higher than that of the first video file. In this way, the basic quality video is obtained after video recording is finished, and the real-time requirement is met; and the basic quality video is further processed in the image library application, so that the user can obtain the video with higher quality on the premise of meeting the real-time requirement.

Description

Technical Field

[0001] The present application relates to the field of image technology, and in particular to a video processing method, an electronic device, and a computer-readable storage medium. Background Art

[0002] With the increasing popularity of electronic devices such as mobile phones, video recording using the video recording function of mobile phones and other electronic devices has gradually become a trend.

[0003] Currently, during the video recording process, mobile phones and other electronic devices first obtain the original image (Raw image) through camera capture, and then use the Image Signal Processing (ISP) to process the Raw image frame by frame in real time to obtain the processed video image; at the end of the video recording, mobile phones and other electronic devices can obtain the final video generated by multiple frames of video images, and the final video is the video captured by the user.

[0004] However, the processing time of the ISP on the Raw image is often subject to real-time constraints, which makes the quality of the video image processed by the ISP low, and further makes the video quality obtained by the user low. Summary of the invention

[0005] The embodiments of the present application provide a video processing method, an electronic device, and a computer-readable storage medium, which can solve the problem of poor video quality of videos captured by existing electronic devices.

[0006] In the first aspect, the embodiment of the present application provides a video processing method, which is applied to an electronic device, and the electronic device is installed with a camera application and a gallery application. After the electronic device displays the video framing interface of the camera application, it receives a first operation acting on the video framing interface and enters video recording; during the video recording process, the original image data and the metadata required by the video algorithm of the second processing process are obtained, and the first processing process is performed on the original image data through the image signal processor to obtain the first video data; if the second operation acting on the video framing interface is received during the video recording process, the video recording is exited, the first video file is obtained, and the first video file and metadata are stored, and the first video file is updated to the gallery application, the first video file is a video file generated by encoding the first video data; the gallery application calls the metadata according to the video algorithm of the second processing process to perform the second processing process on the first video file, obtains the second video file, and stores the second video file; the video quality of the second video file is higher than the video quality of the first video file.

[0007] Among them, the second processing process includes the first type of video algorithm and / or the second type of video algorithm. The first type of video algorithm is a video algorithm that has not been executed in the first processing process, and the second type of video algorithm is a video algorithm that has been executed in the first processing process but the execution effect does not achieve the optimal target effect.

[0008] As can be seen from the above, in the process of video recording, the embodiment of the present application pre-collects the metadata required for the second processing process to be performed by the subsequent gallery application, and performs the first processing process on the original image data in real time through the ISP, so that the first video file can be obtained at the end of the video recording, and the first video file can be viewed by the user, meeting the real-time requirements of the video recording; after obtaining the first video file, the metadata is called on the gallery application side to further perform the second processing process on the first video file to obtain a second video file with higher quality. In this way, the basic quality video (i.e., the first video file) can be obtained at the end of the video recording, meeting the real-time requirements. After the first video file is updated to the gallery application, the gallery application side further executes the video algorithm that has not been executed in the first processing process based on the pre-collected metadata, and the video algorithm that has been executed in the first processing process but the execution effect does not reach the optimal target effect due to real-time and other reasons, so as to improve the video quality of the basic quality video obtained by shooting, and obtain a higher quality video, and then on the premise of meeting the real-time requirements, the user can obtain a higher quality video, thereby improving the user experience.

[0009] In some possible implementations of the first aspect, if the electronic device detects that the camera application is closed or running in the background, or receives a third operation acting on the first thumbnail, the gallery application calls the metadata to perform the second processing on the first video file according to the video algorithm of the second processing process to obtain the second video file. The first thumbnail is a thumbnail of the first video file displayed on the video framing interface after receiving the second operation, that is, when exiting video recording, the thumbnail of the first video file is displayed on the video framing interface.

[0010] In comparison, when video recording is finished but the camera is running in the foreground, if the gallery application calls the metadata to perform the second processing on the first video file, since the camera application still needs to occupy computing resources and memory resources to perform image acquisition and preview video streams and other processing processes, and the gallery application also needs to occupy computing resources and memory resources to run the video algorithm of the second processing process, a large amount of computing resources and memory resources may be occupied, and the processor may be in a high-load state for a long time, which may lead to problems such as device freeze, device heating and high device power consumption, affecting the user experience.

[0011] After the video recording is finished, the camera application is closed, the camera application is running in the background, or a third operation is received on the first thumbnail, the gallery application calls the metadata to perform the second processing on the first video file. Since the electronic device does not need to perform image acquisition and preview video streams and other processing processes at this time, the probability of problems such as device freeze, device heating, and high device power consumption is low, thereby improving the user experience.

[0012] Among them, when the video recording is finished and the camera application is running in the foreground, if the user needs to view the captured video (i.e., receiving the third operation), the electronic device can promptly trigger the gallery application to further process the basic quality video according to the metadata, and promptly obtain a second video file with higher quality, so that the user can promptly view the higher quality video. When the user views the first video file, the camera application jumps to the background and enters the video preview state of the gallery application. At this time, there is no need to perform image acquisition and preview video stream processing.

[0013] In some possible implementations of the first aspect, if the camera application is closed or the camera application is running in the background and a preset trigger condition is detected, the gallery application calls the metadata to perform the second processing process on the first video file according to the video algorithm of the second processing process to obtain the second video file;

[0014] The preset trigger condition includes any one of the following: receiving a fourth operation acting on the second thumbnail or the trigger control, an operation of opening a gallery application, an operation of viewing or playing the first video file, an operation of sharing the first video file, an operation of editing the first video file, the system of the electronic device is in an idle state, and the screen of the electronic device is turned off or the screen is turned off;

[0015] The second thumbnail is a thumbnail of the first video file displayed in the gallery application; the trigger control is a control displayed in the gallery application for triggering video algorithm processing of the first video file.

[0016] In this implementation, when the camera application is running in the background or closed, it further determines whether the preset trigger condition is met. When the preset trigger condition is met, the gallery application calls the metadata to further process the first video file. In this way, when the user needs to view, edit or share the captured video, the gallery application can process the first video file in time, so that the user can obtain a higher quality video in time; in addition, the gallery application can also call the metadata to further process the first video file when the system is idle, the screen is off or the screen is off, which can further alleviate problems such as device freeze and device heating.

[0017] In some possible implementations of the first aspect, receiving a first operation on a video framing interface and entering video recording includes:

[0018] Receive the operation of the video recording mode control on the video framing interface, and determine that the video recording mode is the target recording mode; receive the operation of the recording control on the video framing interface, and enter the video recording of the target recording mode; wherein the first operation includes the operation on the video recording mode control and the operation on the recording control.

[0019] At this time, the video algorithm of the second processing process includes multiple target video algorithms corresponding to the target recording mode. Therefore, when collecting metadata, metadata required by the target video algorithm can be collected. Among the multiple target video algorithms, some target video algorithms require metadata for execution, and some target video algorithms may not require metadata for execution.

[0020] In the target recording mode scenario, the multiple target video algorithms corresponding to the target recording mode have a preset algorithm execution order. At this time, the gallery application executes each target video algorithm in sequence according to the algorithm execution order of the multiple target video algorithms, and when executing multiple target video algorithms, if the target video algorithm requires metadata, the corresponding metadata is called for video algorithm processing. If the target video algorithm does not require metadata, the metadata may not be called for video algorithm processing. By calling the metadata required by the target video algorithm, after executing multiple target video algorithms in sequence, and then performing the encoding operation, the second video file can be obtained; at this time, the video algorithm of the second processing process includes multiple target video algorithms corresponding to the target recording mode, and the target video algorithm can be a first-category video algorithm or a second-category video algorithm.

[0021] In this implementation, based on a video processing framework that executes a first processing process during video recording and a second processing process in a gallery application after video recording is completed, the video recording mode can be enriched, and both video quality and real-time requirements can be taken into account in the target recording mode, so that users can obtain videos with better quality and richer video recording modes, thereby improving the user experience.

[0022] In some possible implementations of the first aspect, the gallery application calls metadata to perform the second processing on the first video file according to the video algorithm of the second processing process to obtain the second video file, including:

[0023] Based on the first video file and the metadata, the gallery application sequentially executes the phased algorithm processing of each target video algorithm according to the algorithm execution order of the multiple target video algorithms corresponding to the target recording mode to obtain a second video file;

[0024] Among them, the phased algorithm processing process of each target video algorithm includes:

[0025] Detect whether a preset trigger condition is met; if the preset trigger condition is met, decode the first target video file to obtain the first target video data; if the current target video algorithm does not require metadata, execute the target video algorithm on the first target video data to obtain the second target video data, and if the current target video algorithm requires metadata, call the metadata required by the current target video algorithm, execute the target video algorithm on the first target video data to obtain the second target video data; encode the second target video data to obtain the second target video file, and store the second target video file;

[0026] Detect whether a preset stop condition is met; if the preset stop condition is not met, enter the staged algorithm processing process of the next target video algorithm; if the preset stop condition is met, determine the second target video file of the current staged algorithm processing process as the second video file; the preset stop condition is that multiple target video algorithms are executed;

[0027] The first target video file is the first video file or the second target video file of the previous staged algorithm processing process, and the first target video data is the video data obtained by decoding the first video data or the second target video file of the previous staged algorithm processing process.

[0028] In this implementation, by executing the second processing procedure in stages, problems such as device freezing and device heating can be further alleviated, thereby improving user experience.

[0029] In some possible implementations of the first aspect, the method further includes: if a preset interruption trigger condition is detected during the staged algorithm processing of the target video algorithm; stop the staged algorithm processing, and store the result data that has been processed in the staged algorithm processing, and when the preset trigger condition is detected, continue to execute the interrupted staged algorithm processing from the interruption point based on the result data; or, stop the staged algorithm processing, store the first target video file of the staged algorithm processing, and re-execute the interrupted staged algorithm processing based on the first target video file when the preset trigger condition is detected. In some possible implementations of the first aspect, the preset interruption trigger condition includes at least one of the following: the electronic device is powered off, the device temperature of the electronic device is higher than the preset temperature threshold, the running memory of the electronic device is lower than the preset memory threshold, and receiving an operation for indicating the termination of processing. In this way, by executing the algorithm processing of each target video algorithm in stages and setting an interruption mechanism, the relevant data can be saved in time when the algorithm processing is interrupted, and all target video algorithms do not need to be re-executed, so as to better implement the second processing process.

[0030] In some possible implementations of the first aspect, the method also includes: when executing each staged algorithm processing process sequentially, if the current staged algorithm processing process is completed, replacing the second target video file of the previous staged algorithm processing process in the storage medium with the second target video file corresponding to the current staged algorithm processing process; the gallery application displays thumbnails of the second target video files of each staged algorithm processing process.

[0031] In this implementation, by saving the processing results of each staged algorithm processing process, the user can view the result of each staged algorithm processing process on the gallery application side, thereby improving the user experience.

[0032] In some possible implementations of the first aspect, the method further includes:

[0033] The gallery application displays a third thumbnail of the first video file;

[0034] The gallery application calls the metadata to perform the second processing on the first video file according to the video algorithm of the second processing process to obtain the second video file, including:

[0035] receiving a fifth operation applied to the third thumbnail, the fifth operation being used to trigger the gallery application to play the first video file;

[0036] In response to the fifth operation, the gallery application decodes the first video file to obtain first video data, where the first video data includes multiple image frames to be processed; for each image frame to be processed, the metadata required by the target video algorithm is called in accordance with the algorithm execution order, and each target video algorithm is executed on the image frame to be processed in turn to obtain the target image frame, and the target image frame is sent for display; wherein, in the process of sequentially executing each target video algorithm, if a target video algorithm does not require metadata, the metadata is not used for video algorithm processing, and if a target video algorithm requires metadata, the corresponding metadata is called for video algorithm processing.

[0037] After each target image frame is sent for display, the gallery application performs encoding operations on each target image frame to obtain a second video file, and replaces the first video file in the storage medium with the second video file.

[0038] In this implementation, when the user needs to play the first video file, the first video file is further processed while the video is playing, and the processed image is displayed in real time to achieve "playing while processing", thereby improving the user experience.

[0039] In a second aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements a method as described in any one of the first aspects above when executing the computer program.

[0040] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements any method as described in the first aspect above.

[0041] In a fourth aspect, an embodiment of the present application provides a chip system, the chip system includes a processor, the processor is coupled to a memory, and the processor executes a computer program stored in the memory to implement any method as described in any one of the first aspects above. The chip system can be a single chip, or a chip module composed of multiple chips.

[0042] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes the method described in the first aspect above.

[0043] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1A A schematic diagram of a video recording scenario provided in an embodiment of the present application;

[0045] Figure 1B A schematic diagram of a video processing process provided in an embodiment of the present application;

[0046] Figure 1C A schematic diagram of the ISP processing process provided in an embodiment of the present application;

[0047] Figure 1D A schematic diagram of the ISP chip processing process provided in an embodiment of the present application;

[0048] Figure 2 A schematic diagram of the structure of an electronic device 200 provided in an embodiment of the present application;

[0049] Figure 3 A schematic block diagram of the video recording system architecture of the electronic device 200 provided in an embodiment of the present application;

[0050] Figure 4 A schematic flow chart of a video processing method provided in an embodiment of the present application;

[0051] Figure 5A A schematic diagram of a video recording process provided in an embodiment of the present application;

[0052] Figure 5B A schematic diagram of a video recording process provided in an embodiment of the present application;

[0053] Figure 5C A schematic diagram of video operation provided in an embodiment of the present application;

[0054] Figure 6 Another schematic block diagram of the video processing method provided in the embodiment of the present application;

[0055] Figure 7 A schematic block diagram of another process of the video processing method provided in the embodiment of the present application;

[0056] Fig. 8A A schematic diagram of a second processing process provided in an embodiment of the present application;

[0057] Figure 8B A schematic diagram of the second processing step in accordance with an embodiment of the present application;

[0058] Figure 8C A schematic diagram of an interrupt handling process provided in an embodiment of the present application;

[0059] Fig.8D A schematic diagram of an interrupt handling process provided in an embodiment of the present application;

[0060] Fig. 9 A schematic block diagram of another process of the video processing method provided in the embodiment of the present application;

[0061] Fig.10 A schematic diagram of the video processing process in the night scene motion mode provided in an embodiment of the present application;

[0062] Fig.11 A schematic diagram of metadata storage provided in an embodiment of the present application. DETAILED DESCRIPTION

[0063] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures and technologies are provided to facilitate a thorough understanding of the embodiments of the present application.

[0064] 1. Video recording (video recording) scene.

[0065] Combine the following Figure 1A and Figure 1B The video recording process is introduced as an example. Figure 1A is a schematic diagram of a video recording scene provided in an embodiment of the present application, Figure 1B It is a schematic block diagram of the video recording system architecture provided in an embodiment of the present application.

[0066] like Figure 1A As shown, the main interface 111 of the mobile phone 11 displays icons such as a camera application 112, a gallery application 113, a calendar application, and a setting application; after the mobile phone 11 receives a click operation on the icon of the camera application 112, it responds to the click operation and displays a video preview interface 114 of the camera application 112. The video preview interface 114 includes a viewfinder and controls 115, etc., and the viewfinder displays the picture captured by the camera of the mobile phone 11.

[0067] After receiving the click operation on the control 115 , the mobile phone 11 enters the video recording in response to the click operation and displays the video recording interface 116 . The video recording interface 116 includes the video screen being recorded and the control 117 .

[0068] After receiving the click operation on the control 117, the mobile phone 11 exits the video recording in response to the click operation, obtains the recorded video, displays a thumbnail 118 of the recorded video on the video preview interface, and updates the recorded video to the gallery application 113. The user can jump to the gallery application 113 to view the recorded video by clicking the thumbnail 118, or enter the gallery application 113 to view the recorded video by clicking the icon of the gallery application 113.

[0069] like Figure 1B As shown, the video recording system framework includes: an application layer, an intermediate layer and a functional module layer. The application layer includes a camera application and a gallery application. The intermediate layer includes a camera service and a media library. The camera service includes a real-time video recording sub-service. The real-time video recording sub-service includes camera devices and camera streams, etc. The real-time video recording sub-service is used to implement the video recording function. The media library includes a Media Store database interface and a Media Provider-DB (Media Provider database). The functional module layer includes an acquisition module, an algorithm processing module and a file system, etc. The acquisition module is used to acquire image data. The algorithm processing module is used to execute multiple algorithm processes to perform real-time online processing of image data. The file system is used to store recorded videos, which are usually stored in non-volatile storage media, such as a read-only memory (ROM).

[0070] exist Figure 1AIn the scenario shown, after the mobile phone 11 receives a click operation on the control 115, the camera application initiates a video recording request to the camera service. In response to the video recording request, the camera service calls the acquisition module to perform image acquisition, obtains the image data to be processed, and stores the image data to be processed in the image buffer (image buff), and performs algorithm processing on the image data to be processed in real time through the algorithm processing module. In the algorithm processing stage, multiple algorithm processes (process, proc) are sequentially executed on the image data to be processed to obtain a video stream in real time, and the video stream is returned to the camera application through the camera service.

[0071] After the mobile phone 11 receives the click operation on the control 117, the video encoding module encapsulates and encodes the video image output by the algorithm processing module to obtain the recorded video when the video recording ends, and updates the recorded video to the gallery application. The gallery application can access the recorded video in the file system through the media library. Figure 1B FIG. 4 exemplifies that the recorded video is a 100-point basic quality video.

[0072] 2. Video processing process.

[0073] In the video recording scenario, the electronic device processes the image captured by the camera in real time online through the ISP chip, so that the recorded video can be obtained when the video recording ends, ensuring the real-time nature of the video recording. The processing process of the ISP chip may include: Figure 1B The process of the algorithm processing module in .

[0074] For example, see Figure 1C The video processing process diagram provided by the embodiment of the present application is shown, based on Figure 1A In the video recording scene shown, the camera devices such as the lens and image sensor (Sensor) of the mobile phone 11 capture the light signal of the framing environment, convert the light signal into an electrical signal, and output a Raw image (i.e., original image data); the ISP chip executes multiple processing algorithms in real-time serially to process each frame of the Raw image in real time online to obtain a processed image; the processed image output by the ISP chip can be divided into two data streams, one data stream is a preview video stream, and the other data stream is a recorded video stream. The preview video stream refers to the video stream displayed on the display screen of the electronic device during the video recording process, and the recorded video stream refers to the video stream that is saved in the electronic device after the video recording is completed and can be viewed by the user. As shown Figure 1B As shown, after the encoding module performs video encoding on the processed image output by the ISP chip, a preview video stream is obtained, and the preview video stream is sent for display to display the video screen on the display screen, for example, Figure 1AThe video preview interface 114 and the video recording interface 116 display the video screen; in addition, the encoding module of another channel performs video encoding and other operations on the processed image output by the ISP chip to generate a video file, and stores the video file in a storage medium, and updates the video file to the gallery application 113, so that the user can view the video file in the gallery application 113.

[0075] In the related art, after the video file is updated to the gallery application 113, the gallery application 113 only supports viewing, editing and sharing of the video file, and does not further process the video file. After the captured video is placed in the gallery application 113 for a period of time, the video quality does not change.

[0076] The ISP chip executes multiple processing algorithms in a pipelined manner to improve image effects such as details, color, and brightness. For example, see Figure 1D The schematic diagram of the ISP chip processing process provided by the embodiment of the present application is shown. After the Raw image is input into the ISP, it is processed by the processing algorithms such as bad pixel correction, black level correction, noise reduction, lens brightness correction, automatic white balance, color interpolation, color correction, Ygamma, sharpening and color space conversion in sequence to obtain a YUV image. The YUV image is processed by video encoding and packaging, etc., to obtain a video file.

[0077] It is understood that the ISP chip may include Figure 1D More or fewer processing algorithms. For example, the ISP chip can also include anti-shake algorithm and beauty algorithm.

[0078] The inventors have discovered during long-term research that video recording in related technologies cannot take into account both real-time performance and video quality, and usually sacrifices video quality to ensure real-time performance of video recording.

[0079] Specifically, in order to ensure the real-time recording of the video, so that the user can obtain the recorded video when the video recording ends, the processing time of each frame of the Raw image is often subject to real-time constraints, which makes the ISP only run basic processing algorithms to process the Raw image, and the algorithm effect of each processing algorithm is sacrificed to meet the demand for real-time video output. In this way, although the real-time requirements of video recording can be met, the video quality of the obtained video file is poor. For example, in Figure 1AIn the scenario shown, when the user ends the video recording by clicking the control 117, the mobile phone 11 needs to generate a video file in real time when the video recording ends for the user to view. At this time, when the user records a video at 30fps (Frame Per Second), the processing time of each frame of the Raw image may be around 33 milliseconds. Within 33 milliseconds, the number and types of image processing algorithms that the ISP can execute are limited, and the algorithm effects of some image processing algorithms will also be sacrificed, making the execution effect of the image processing algorithm very limited, and thus making the video quality of the final video file poor.

[0080] In response to the related issues mentioned above, an embodiment of the present application provides a video processing solution, which processes the original image data in real time on the ISP chip side during the video recording process, so as to obtain a first video file in real time when the video recording ends, thereby meeting the real-time requirements of video recording; after the video recording ends, the gallery application side further processes the first video file based on metadata to obtain a second video file with higher quality.

[0081] Different from executing the processing algorithm serially in real time online only through the ISP chip side during the video recording process, the video processing solution of the embodiment of the present application not only executes the processing algorithm serially in real time online through the ISP chip during the video recording process, but also performs further offline processing on the captured video file based on metadata at the gallery application side after the video recording is finished, so as to obtain a higher quality video file. In this way, the basic quality video (i.e., the first video file) can be obtained at the end of the video recording, meeting the real-time requirements of the video recording; after the video recording is finished, the basic quality video captured is further processed offline based on the pre-collected metadata at the gallery application side to obtain a higher quality video, thereby enabling users to obtain higher quality videos while meeting the real-time requirements, thereby improving the user experience.

[0082] The video processing solution provided in the embodiment of the present application can be applied to an electronic device, which may be, for example, a mobile phone or a tablet computer, etc., and the type of the electronic device is not limited here.

[0083] For example, see Figure 2 The structural diagram of the electronic device 200 is shown. The electronic device 200 may include but is not limited to: a processor 210, a memory 220, a camera 230, and a display screen 240, etc.

[0084] It is understandable that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 200. In other embodiments of the present application, the electronic device 200 may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware. For example, when the electronic device 200 is a mobile phone, the electronic device 200 may also include: a universal serial bus (USB) interface, a charging management module, a sensor module, a power management module, a battery, an antenna, a mobile communication module, a wireless communication module, an audio module, a speaker, a receiver, a microphone, a headphone interface, a button, a motor, an indicator, and a subscriber identification module (SIM) card interface; the sensor module may include at least one of the following: a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a temperature sensor, a touch sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, an ambient light sensor, and a bone conduction sensor, etc.

[0085] The processor 210 may include one or more processing units, for example, the processor 210 may include an application processor (AP), a graphics processing unit (GPU), an ISP, a controller, a video codec, a digital signal processor (DSP), etc. Among them, different processing units may be independent devices or integrated into one or more processors. The controller may generate an operation control signal based on the instruction opcode and the timing signal to complete the control of fetching and executing instructions. It can be understood that when the gallery application of the embodiment of the present application calls metadata to further process the first video file, one or more processors such as AP, GPU, ISP and DSP may be involved. At this time, the ISP is processed offline.

[0086] In some embodiments, the processor 210 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, etc.

[0087] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 210 may include multiple I2C buses. The processor 210 may be coupled to the camera 230, etc. through different I2C bus interfaces.

[0088] The MIPI interface can be used to connect the processor 210 with peripheral devices such as the display screen 240 and the camera 230. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 210 and the camera 230 communicate via the CSI interface to implement the shooting function of the electronic device 200. The processor 210 and the display screen 240 communicate via the DSI interface to implement the display function of the electronic device 200.

[0089] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 210 with the camera 230, the display screen 240, etc. The GPIO interface can also be configured as an I2C interface and a MIPI interface, etc.

[0090] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 200. In other embodiments of the present application, the electronic device 200 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0091] The electronic device 200 implements the display function through a GPU, a display screen 240, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 240 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 210 may include one or more GPUs, which execute program instructions to generate or change display information.

[0092] The display screen 240 is used to display images and videos, etc. The display screen 240 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 200 may include 1 or N display screens 240, where N is a positive integer greater than 1.

[0093] The electronic device 200 can realize the shooting function through the ISP, the camera 230, the video codec, the GPU, the display screen 240 and the application processor.

[0094] The ISP is used to process the data fed back by the camera 230. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. The ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP can be set in the camera 230.

[0095] The camera 230 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 200 may include 1 or N cameras 230, where N is a positive integer greater than 1.

[0096] The digital signal processor is used to process the digital image signal.

[0097] Video codecs are used to compress or decompress digital videos. The electronic device 200 may support one or more video codecs. Thus, the electronic device 200 may play or record videos in a variety of coding formats, such as moving picture experts group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0098] The memory 220 can be used to store computer executable program codes, which include instructions. The memory 220 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 200 (such as audio data, a phone book, etc.), etc. In addition, the memory 220 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 210 executes various functional applications and data processing of the electronic device 200 by running instructions stored in the memory 220, and / or instructions stored in a memory provided in the processor.

[0099] After introducing the possible hardware architecture of the electronic device 200 according to the embodiment of the present application, Figure 3 The schematic block diagram of the video recording system architecture of the electronic device 200 is shown to exemplarily introduce the software system architecture of the electronic device 200 .

[0100] like Figure 3 As shown, the software system architecture of the electronic device 200 includes an application layer, a framework and service layer, and a hardware abstract layer (HAL).

[0101] The application layer includes the camera application and the gallery application. The camera application can implement the video recording function and the video viewing function. The gallery application can access the video and metadata (Meta data) in the file system through the media library.

[0102] The framework and service layer include camera service and media library. Camera service includes real-time video recording sub-service, and real-time video recording sub-service may include camera device and camera stream, etc. Media library includes modules such as Media Store database interface, MediaProvider-DB, interception trigger module and segmented video startup service. Segmented video startup service includes functional modules such as scheduling strategy and task scheduling. Segmented video startup service is used to schedule algorithm tasks according to scheduling strategy. Algorithm tasks may refer to each algorithm process (proc) in segmented algorithm processing. Each algorithm process is scheduled according to scheduling strategy to execute each algorithm process in order. The interception trigger module is used to detect trigger conditions, and when the trigger conditions are detected, the segmented startup service is triggered to perform algorithm task scheduling to schedule the execution of multiple algorithm processes to obtain 120 points of high-quality video. The trigger condition may be, for example, a user clicking on the thumbnail of a video file, or a user viewing, sharing, or editing a video file, etc.

[0103] HAL includes camera HAL, algorithm service, segmented algorithm service and file system. Camera HAL includes an acquisition module, which is used to acquire image data. Algorithm service is used to execute multiple algorithm processing to process image data in real time online to obtain 100-point basic quality video. Segmented algorithm service includes sub-modules such as algorithm scheduling management, segmented algorithm processing 1 and segmented processing algorithm N. Algorithm scheduling management is used to schedule and manage each processing algorithm. Segmented algorithm processing 1 and segmented processing algorithm N both include multiple algorithm processes (proc). Metadata, 100-point basic quality video and 120-point high-quality video are stored in the file system.

[0104] It is worth pointing out that in Figure 3 In the video recording system architecture shown in FIG. 1 , the video processing process is exemplarily divided into two processes, stage 1 and stage 2. Basic quality video is generated in stage 1, and high quality video is generated in stage 2. In addition, stage 2 is also divided into multiple segmented algorithm processing processes (such as Figure 3 This is just an example and does not limit the technical solution of the present application. For example, in practical applications, it is not necessary to divide stage 2 into multiple segmented algorithm processing processes.

[0105] Different from Figure 1B The system architecture, Figure 3 The system architecture adds related modules for algorithm processing of 100-point basic quality videos on the gallery application side, such as interception trigger module and segmented algorithm processing 1 to N. For users, the quality of the video will improve after it is placed in the gallery application for a period of time.

[0106] It is understandable that after obtaining the 120-point high-quality video, the electronic device 200 can also delete the Meta data and the 100-point basic quality video stored in the file system, and change the thumbnails and other controls originally associated with the 100-point basic quality video to be associated with the 120-point high-quality video.

[0107] In addition, the 100-point basic quality video and 120-point high-quality video are only for the convenience of expression, so as to show that the video quality of stage two is higher than that of stage one, and will not limit the actual video quality of the video files shot in stage one and the video files shot in stage two.

[0108] The video processing solution provided in the embodiment of the present application is further described below in combination with the hardware system architecture and software system architecture of the electronic device 200.

[0109] See also Figure 4 A schematic flow chart of a video processing method provided by an embodiment of the present application is shown, and the video processing method may include:

[0110] Step S401: The electronic device 200 displays a video viewing interface of a camera application.

[0111] It is understandable that after receiving the click operation acting on the camera application, the electronic device 200 can start the camera application in response to the click operation and display the video framing interface of the camera application. At this time, the electronic device 200 is in the video preview state and has not entered the video recording process.

[0112] For example, see Figure 5A The schematic diagram of the video recording process provided by the embodiment of the present application is shown, and application icons such as a camera application 53, a gallery application 54, and a setting application are displayed on the main interface 52 of a mobile phone 51. When the mobile phone 51 receives a click operation on the camera application 53, the mobile phone 51 responds to the click operation and displays a video framing interface 55. The framing frame of the video framing interface 55 displays a preview video stream and also includes a video recording control 56 and more controls 57.

[0113] Step S402: the electronic device 200 receives a first operation on the video framing interface and enters video recording.

[0114] The first operation is an operation for triggering the electronic device 200 to enter a video recording process, which may include one or more operations.

[0115] In some embodiments, the first operation refers to an operation on a video recording control to trigger the electronic device 200 to enter a video recording process in a normal mode.

[0116] In other embodiments, the first operation may include an operation on a video recording module control and an operation on a video recording control. The operation on the video recording module control is used to trigger the electronic device 200 to determine a target recording mode, and the operation on the video recording control triggers the electronic device 200 to enter a video recording process in the target recording mode. The target recording mode refers to a specific video recording mode, which is different from an ordinary video recording mode. Exemplarily, the target recording mode may include, but is not limited to, a movie mode, a telephoto mode, a night scene sports mode, and a slow motion mode.

[0117] For example, Figure 5A As shown, after the mobile phone 51 displays the video framing interface 55, the user can click the video recording control 56 to trigger the mobile phone 51 to enter the video recording process in the normal mode; or click the more controls 57 to select the video recording mode and enter the video recording process in a specific video recording mode.

[0118] When the user clicks the video recording control 56 on the video recording view interface 55, the mobile phone 51 enters the video recording in response to the click operation on the video recording control 56, and displays the video recording view interface 58 in the video recording state. At this time, the first operation is the click operation on the video recording control 56.

[0119] When the user clicks on the more controls 57 on the video framing interface 55, the mobile phone 51 displays a window 59. The window 59 includes a variety of video recording modes such as movie mode, telephoto mode, night scene motion mode and slow motion mode. When the user clicks on the night scene motion mode control displayed in the window 59, the mobile phone 51 responds to the click operation, determines that the target recording mode is the night scene motion mode, and displays the video framing interface 510 in the video preview state of the night scene motion mode; after the mobile phone 51 receives the click operation on the video recording control 56 on the video framing interface 510, it responds to the click operation, enters the video recording of the night scene motion mode, and displays the video framing interface 58 in the video recording state. At this time, the first operation may include a click operation on the more controls 57, a click operation on the night scene motion mode in the window 59, and a click operation on the video recording control 56.

[0120] Step S403: During the video recording process, the electronic device 200 obtains the original image data and metadata required by the video algorithm of the second processing process, and performs the first processing process on the original image data through the image signal processor to obtain the first video data.

[0121] The original image data (i.e., Raw image) refers to the image data output by the image sensor. The Raw image output may be different depending on the pixel array of the image sensor. Usually, the pixel array of the image sensor adopts the Bayer array, and the Raw image is usually a Bayer image.

[0122] It should be pointed out that during the video recording process, in addition to collecting 3A (automatic exposure control AE, automatic focus control AF, automatic white balance AWB) information, image information (such as image width and height), and exposure parameters (aperture size, shutter speed and sensitivity aperture value) and other metadata (Mate data), the electronic device 200 also specifically collects the metadata required for the subsequent gallery application to execute the video algorithm of the second processing process.

[0123] Normally, when the electronic device 200 enters the video recording process of a specific video recording mode, the electronic device 200 can specifically collect metadata of the video algorithm corresponding to the specific video recording mode. The video algorithms required to be executed in various specific video recording modes are preset, and the execution order between multiple video algorithms can also be preset. For example, the video algorithm corresponding to the night scene motion mode includes a noise reduction algorithm, an anti-shake algorithm, and a deblurring algorithm. At this time, when the electronic device 200 enters the video recording process of the night scene motion mode, the metadata collected by the electronic device 200 includes gyroscope (Gyro) information and a low fps Raw image. Gyroscope information can be used for a deblurring algorithm and an anti-shake algorithm, and a low fps Raw image can be used for a noise reduction algorithm. At this time, the algorithm of the second processing process includes multiple video algorithms corresponding to a specific video recording mode.

[0124] When the electronic device 200 enters the video recording process in the normal mode, the electronic device 200 can first perform scene recognition based on the Raw image, and collect the required metadata based on the recognized scene. For example, when the electronic device 200 can recognize that the current scene is a night scene based on the raw image data, the metadata required for the night scene (such as exposure data, etc.) can be collected in a targeted manner.

[0125] It is understandable that the video algorithms of the second processing process include multiple video algorithms, but not all video algorithms require metadata, that is, in the second processing process, some video algorithms need to use metadata, while some video algorithms do not need metadata. Therefore, when collecting metadata, the metadata required by the video algorithms that need to use metadata is collected.

[0126] It is understandable that after the electronic device 200 enters the video recording, the Raw image output by the image sensor is processed in real time online by the video ISP chip to obtain the first video data, that is, the ISP performs a first processing process on the Raw image to obtain the first video data. The first processing process generally includes running multiple video algorithms in the video ISP chip to process the Raw image.

[0127] Step S404: If the electronic device 200 receives a second operation on the video framing interface during the video recording process, it exits the video recording, obtains the first video file, stores the first video file and metadata, and updates the first video file to the gallery application. The first video file is a video file generated by encoding the first video data.

[0128] The second operation is an operation for triggering the electronic device 200 to exit video recording, and the operation may include one operation or multiple operations.

[0129] For example, see Figure 5B A schematic diagram showing the video recording process, based on Figure 5A In the scenario, after the mobile phone 51 enters the video recording and displays the video framing interface 58 in the video recording state, the mobile phone 51 continuously performs video recording. During the continuous video recording process, when the mobile phone 51 receives a click operation on the video recording control 511 on the video framing interface 58, the mobile phone 51 exits the video recording in response to the click operation, generates a first video file, and stores the first video file and metadata in a non-volatile storage medium. At this time, the second operation is a click operation on the video recording control 511.

[0130] Among them, the video ISP chip processes each frame of the Raw image in real time online. After obtaining the first video data corresponding to each frame of the Raw image, the electronic device 200 can encode and package each first video data to generate a first video file when the video recording ends.

[0131] After the electronic device 200 updates the first video file to the gallery application, the gallery application displays a thumbnail of the first video file, and the user can click on the thumbnail of the first video file to view, play, edit, and share the first video file. Generally, the electronic device 200 can insert a file record of the first video file in the media library, and according to the file record of the first video file, the gallery application can read the first video file from the file system, thereby updating the first video file to the gallery application.

[0132] Optionally, the electronic device 200 may also display a thumbnail of the first video file on the video viewing interface. Figure 5BAfter receiving the click operation on the video recording control 511, the mobile phone 51 exits the video recording in response to the click operation, and displays the video viewing interface 512 in the video preview state, and displays the thumbnail 513 of the first video file on the video viewing interface 512. When the user clicks the thumbnail 513 of the first video file, the electronic device 200 jumps from the camera application to the gallery application and displays the first video file.

[0133] See also Figure 2 and Figure 3 , the first video file may refer to a 100-point basic quality video, and steps S401 to S404 may refer to stage one, i.e., the process of generating a basic quality video. In the process of generating a basic quality video, when the electronic device 200 receives the first operation, the camera application responds to the first operation and sends a video request to the real-time video recording sub-service; the real-time video recording sub-service responds to the video recording request, calls the camera 230 through the acquisition module of the camera HAL to perform image acquisition, obtains the Raw image output by the image sensor, and specifically acquires the metadata required by the video algorithm of stage two; the Raw image output by the acquisition module is transmitted to the algorithm processing module of the algorithm service, and the algorithm processing module performs multiple algorithm processes (i.e., video algorithms) on the Raw image in real time and sequentially to obtain the first video data; when the electronic device 200 receives the second operation, it exits the video recording, performs an encoding operation on the first video data, generates a first video file (i.e., a 100-point basic quality video), and updates the 100-point basic quality video to the gallery application through the media library, and stores the 100-point basic quality video and metadata to the file system. At this point, Phase 1 is completed, a 100-point basic quality video is obtained, and the 100-point basic quality video is updated to the Gallery application.

[0134] It is worth pointing out that during the video recording process, the electronic device 200 processes the Raw image online in real time through the video ISP chip, and can obtain a 100-point basic quality video when the video recording ends, thus meeting the real-time nature of video recording.

[0135] In addition, compared to not generating a video file when ending video recording, the embodiment of the present application, when ending video recording, encodes and packages the processing results of the first processing process to generate a first video file. The first video file can be viewed by the user, meeting the real-time requirements of video recording and providing a higher user experience.

[0136] Step S405, the gallery application of the electronic device 200 calls the metadata to perform the second processing on the first video file according to the video algorithm of the second processing process, obtains the second video file, and stores the second video file; the video quality of the second video file is higher than the video quality of the first video file.

[0137] Among them, the second processing process includes the first type of video algorithm and / or the second type of video algorithm. The first type of video algorithm is a video algorithm that has not been executed in the first processing process, and the second type of video algorithm is a video algorithm that has been executed in the first processing process but the execution effect does not achieve the optimal target effect.

[0138] After the electronic device 200 updates the first video file to the gallery application, the gallery application can directly perform further processing on the first video file. For example, when the camera application is running in the foreground, the gallery application performs the second processing process on the first video file in the background; of course, the gallery application can also perform further processing on the first video file at an appropriate time. The appropriate time may refer to: detecting that the camera application is closed or the camera application is running in the background, receiving the third operation acting on the first thumbnail, or the camera application is closed or the camera application is running in the background, and the preset trigger condition is met. The first thumbnail is a thumbnail of the first video file displayed on the video framing interface after receiving the second operation.

[0139] For example, see Figure 5C The video operation diagram is shown, based on Figure 5B In the scenario, the mobile phone 51 exits the video recording, and displays the thumbnail 513 of the first video file on the viewfinder interface 512 in the video preview state. At this time, the camera application is running in the foreground, and the mobile phone 51 needs to perform processes such as image acquisition and preview video stream processing; after the mobile phone receives the click operation on the thumbnail 513 of the first video file, it jumps to the video viewing interface 514 of the gallery application, and the video viewing interface 514 includes a sharing control 515, an editing control 516, and a trigger control 517. At this time, the camera application is running in the background, and the gallery application is running in the foreground, and the mobile phone 51 does not need to perform processes such as preview video stream processing. The first thumbnail is thumbnail 513, and the third operation is a click operation on thumbnail 513.

[0140] In comparison, when the video recording is finished but the camera is running in the foreground, the camera application still needs to occupy computing resources and memory resources to perform processing processes such as image acquisition and previewing video streams. If the gallery application executes multiple video algorithms of the second processing process to process the first video file at this time, the multiple video algorithms also need to occupy computing resources and memory resources. Therefore, the computing resources and memory resources of the electronic device 200 may be highly loaded, and the processor may be in a high-load state for a long time, which may cause the electronic device 200 to experience problems such as freezing, heating, and high power consumption, affecting the user experience.

[0141] After the video recording is finished, the camera application is closed, the camera application is running in the background, or a third operation is received on the first thumbnail, the gallery application processes the first video file. Since the electronic device does not need to perform image acquisition and preview video streams and other processing processes at this time, the electronic device 200 is less likely to experience problems such as freezing, heat, and high power consumption, thereby improving the user experience.

[0142] Furthermore, when the camera application is closed or running in the background, the electronic device 200 determines whether a preset trigger condition is detected. If the preset trigger condition is detected, the gallery application is triggered to call the metadata to process the first video file to obtain the second video file; if the preset trigger condition is not detected, it continues to determine whether the preset trigger condition is detected.

[0143] Among them, the preset trigger conditions include any one of the following: receiving a fourth operation acting on a second thumbnail or a trigger control, an operation of opening a gallery application, an operation of viewing or playing a first video file, an operation of sharing a first video file, an operation of editing a first video file, the system of the electronic device is in an idle state, and the screen of the electronic device is turned off or on.

[0144] The trigger control is a control displayed in the gallery application for triggering the video algorithm processing of the first video file. For example, see Figure 5C The video viewing interface 514 of the mobile phone 51 includes a trigger control 517; when the mobile phone 51 receives a click operation (ie, the fourth operation) acting on the trigger control 517, it is considered that a preset trigger condition is detected, and the gallery application is triggered to further process the first video file.

[0145] The second thumbnail is the thumbnail of the first video file displayed in the gallery application. Figure 5C After receiving the click operation on the gallery application 54, the mobile phone 51 displays the album interface in response to the click operation, and the album interface includes the video album control 519. After receiving the click operation on the video album control 519, the mobile phone 51 displays the video album interface 520. The video album interface 520 includes thumbnails of each video. Figure 5C exemplarily shows thumbnails 521 to 525, wherein thumbnail 524 is a thumbnail of the first video file, that is, the second thumbnail. After receiving the click operation (i.e., the fourth operation) on thumbnail 524, mobile phone 51 displays video viewing interface 514, and considers that the preset trigger condition is detected, triggering the gallery application to process the first video file.

[0146] Please see again Figure 5CThe operation of opening the gallery application refers to the operation of clicking on the gallery application 54; the operation of viewing or playing the first video file refers to the operation of clicking on the first video file displayed on the video viewing interface 514; the operation of sharing the first video file refers to the operation of clicking on the sharing control 515 in the video viewing interface 514; the operation of editing the first video file refers to the operation of clicking on the editing control 516 in the video viewing interface 514.

[0147] The system of the electronic device 200 being in an idle state may mean that: the load of the processor is lower than a preset load threshold, and the running memory is higher than a preset memory threshold. Generally, when the user is not using the electronic device 200, and no application is running in the foreground of the electronic device 200, and no application is resident in the background, it can be considered that the electronic device is in an idle state, which can be specifically judged by the load of the processor, the running memory, and the device temperature.

[0148] Of course, the preset trigger conditions are not limited to those mentioned above. For example, the preset trigger conditions may also include: uploading the first video file to the cloud, or the processor load of the electronic device 200 is lower than the preset load threshold, the running memory is higher than the preset memory threshold, and the device temperature is lower than the preset temperature threshold.

[0149] It is worth pointing out that when the camera application is running in the background or is closed, it further determines whether the preset trigger conditions are met. When the preset trigger conditions are met, the gallery application further processes the first video file. In this way, when the user needs to view, edit or share the captured video, the gallery application can process the first video file in time, so that the user can obtain a higher quality video in time; in addition, the gallery application can also further process the first video file when the system is idle, the screen is off or the screen is off, which can further alleviate problems such as device freezes and device heating.

[0150] Generally, after the first video file is captured, if the user needs to operate the first video file (for example, edit, view, share or upload to a third-party platform, etc.), the electronic device 200 can promptly trigger the gallery application to further process the first video file to promptly obtain a second video file with higher quality. If the user does not need to operate the first video file, the electronic device 200 can trigger the gallery application to further process the first video file when the system is idle or the total system load is low, so as to alleviate device freezes and device heating and reduce device power consumption.

[0151] In the process of the gallery application calling the metadata to process the first video file, it is usually necessary to execute multiple video algorithms. That is, the second processing process may include multiple video algorithms. The multiple video algorithms may be divided into multiple stages, each stage including one or more video algorithms. For example, see Figure 3 The segmented algorithm service includes segmented algorithm processing 1 to segmented algorithm processing N, and each segmented algorithm processing includes multiple procs (ie, video algorithms).

[0152] The second video file may refer to Figure 3 In the 120-point basic quality video, step S405 may refer to stage two, i.e., the process of generating high-quality video. In the process of generating high-quality video, when the interception trigger module detects a trigger condition (e.g., when viewing, sharing or editing a video file), the segmented video startup service is triggered to start the segmented algorithm processing. The segmented video startup service performs task scheduling on the algorithm processing process of stage two, and after determining the scheduling strategy according to the system state, each segmented algorithm processing of stage two is executed using the determined scheduling strategy. After multiple video algorithms are executed, the processed video data can be encoded and packaged to obtain a 120-point high-quality video (i.e., the second video file), and the 120-point high-quality video (i.e., the second video file) is updated to the gallery application through the media library, and the 120-point high-quality video (i.e., the second video file) is stored in the file system. The quality of the 120-point high-quality video is higher than the quality of the 100-point basic quality video.

[0153] It should be noted that when storing the second video file, the first video file in the original storage medium can be replaced by the second video file, or the second video file can be directly stored in the storage medium. The first video file and the second video file are in the storage medium at the same time, but in different storage locations.

[0154] It is worth pointing out that the embodiment of the present application generates a 100-point basic quality video in real time through stage one to meet the real-time requirements of video recording; further, a 120-point high-quality video is generated through stage two, so that users can obtain higher quality video files while meeting the real-time requirements. It can be understood that stage one has strong real-time requirements, so the video ISP chip usually only executes basic processing algorithms, and sacrifices the algorithm execution effect to meet the needs of real-time video output, thereby obtaining a 100-point basic quality video; the second processing process (stage two) has a weaker real-time requirement, and multiple video algorithms can be further run, and the execution effect of the video algorithm can be executed to the optimal target effect to improve the video quality of the 100-point basic quality video.

[0155] During the video recording process, the embodiment of the present application pre-collects metadata required by the subsequent gallery application, and processes the original image data through the ISP in real time, so that the basic quality video (i.e., the first video file) can be obtained at the end of the video recording, meeting the real-time requirement; after the first video file is updated to the gallery application, the gallery application further processes the basic quality video obtained by shooting according to the pre-collected metadata to obtain a higher quality video, thereby allowing users to obtain higher quality videos while meeting the real-time requirement, thereby improving the user experience.

[0156] Based on the above embodiments, the video processing process can be divided into two stages. The first stage refers to the process of generating a first video file, and the second stage refers to the process of generating a second video file based on the first video file. In the first stage, the video ISP chip executes multiple video algorithms online in real time; in the second stage, the gallery application executes multiple video algorithms offline. Figure 6 Another schematic flow chart of the video processing method is shown in FIG. Figure 6 As shown, the video processing method may include the following steps:

[0157] Step S601: The electronic device 200 displays a video viewing interface of a camera application.

[0158] Step S602: The electronic device 200 receives a first operation on the video viewing interface and enters video recording.

[0159] Step S603: During the video recording process, the electronic device 200 obtains the original image data and metadata required by the video algorithm of the second processing process, and performs the first processing process on the original image data through the image signal processor to obtain the first video data. The first processing process includes multiple video algorithms.

[0160] The first processing step may refer to Figure 3 The multiple procs (algorithm processes) executed in the algorithm processing module are processed in real time online on the video ISP chip side, and the real-time constraints are relatively high. In order to meet the real-time constraints, the number of video algorithms that can be executed by the first processing process and even the execution effect of the video algorithm are subject to real-time constraints. Therefore, the multiple video algorithms executed by the first processing process are usually relatively basic video algorithms, such as noise reduction algorithms and sharpening algorithms.

[0161] In the embodiment of the present application, the first processing process performed by the video ISP chip side can be Figure 1B The algorithm processing shown is the same, that is, the first processing process of the embodiment of the present application may not reduce the original Figure 1BBecause reducing the video algorithm executed by the first processing process will reduce the video quality of the first video file, affecting the user experience.

[0162] Step S604: If the electronic device 200 receives a second operation on the video framing interface during the video recording process, it exits the video recording, obtains the first video file, stores the first video file and metadata, and updates the first video file to the gallery application. The first video file is a video file generated by encoding the first video data.

[0163] It is worth noting that compared with not generating a video file when ending video recording, the embodiment of the present application, when ending video recording, encodes and packages the processing results of the first processing process to generate a first video file. The first video file can be viewed by the user, meeting the real-time requirements of video recording and providing a higher user experience.

[0164] Step S605: When the electronic device 200 detects that the camera application is closed or running in the background and receives a third operation on the first thumbnail, or the camera application is closed or running in the background and a preset trigger condition is met, the gallery application calls the metadata to perform the second processing on the first video file according to the video algorithm of the second processing process, obtains the second video file, and stores the second video file; the second processing process includes a first type of video algorithm and / or a second type of video algorithm, the first type of video algorithm is a video algorithm that has not been executed in the first processing process, and the second type of video algorithm is a video algorithm that has been executed in the first processing process but the execution effect does not reach the optimal target effect; the video quality of the second video file is higher than the video quality of the first video file.

[0165] It is understandable that metadata includes data required by multiple video algorithms. When metadata is called to perform the second processing on the first video file, if a certain video algorithm needs to use metadata, the metadata required by the video algorithm is called to perform video algorithm processing, and if a certain video algorithm does not need to use metadata, metadata does not need to be called when executing the video algorithm.

[0166] As mentioned above, when the first processing process executes the video algorithm, due to the high real-time constraints, the execution effect of some video algorithms cannot be the best, and the number of video algorithms that can be executed is also very limited, which makes the quality of the first video file poor. In order to improve the quality of videos shot by users while meeting the real-time requirements, the second processing process is executed offline on the gallery application side. The real-time constraints of the second processing process are usually low. The second processing process may include video algorithms that have been executed in the first processing process but the execution effect did not reach the target effect, that is, the second type of video algorithm, and may also include video algorithms that have not been executed in the first processing process, that is, the first type of video algorithm. Both the first type of video algorithm and the second type of video algorithm can improve video quality.

[0167] For example, the first processing process may include algorithms such as bad pixel correction, black level correction, noise reduction, lens brightness correction, automatic white balance, color interpolation, color correction, Ygamma, sharpening, and color space conversion; the second processing process may include algorithms such as anti-shake, noise reduction, and deblurring. In this case, the anti-shake algorithm and the deblurring algorithm are the first type of video algorithms, and the noise reduction is the second type of video algorithm.

[0168] Among them, the above-mentioned optimal target effect refers to the video picture effect that is desired to be achieved. Taking the noise reduction algorithm as an example, the optimal target effect may refer to: the video picture has less noise and rich image details. When the first processing process executes the noise reduction algorithm, due to real-time constraints, usually only part of the noise in the image is removed, but a large number of image details lost during noise removal are not restored, resulting in the loss of image details in the obtained video picture. When the second processing process executes the noise reduction algorithm, not only the remaining noise in the image can be removed, but also the image details lost during denoising can be restored, thereby achieving the optimal target effect of less noise in the video picture and rich image details, thereby improving the quality of the video picture.

[0169] It is worth pointing out that the embodiment of the present application does not put the video algorithm originally belonging to the first processing process into the subsequent second processing process for execution, but does not reduce the video algorithm of the first processing process, and further executes the required video algorithm in the second processing process to improve the video quality.

[0170] Of course, in other embodiments, the video algorithms executed in the first processing process may be appropriately reduced, and the reduced video algorithms may be executed in the second processing process. However, in this case, the video algorithms originally belonging to the first processing process are not simply executed in the second processing process, but the second processing process also executes the video algorithms that were not executed in the first processing process or the execution effect is not the best, so as to improve the video quality.

[0171] See also Figure 3The first processing process may refer to multiple procs executed in the algorithm processing module in stage one, and the second processing process may refer to the segmented algorithm processing 1 to N in the segmented algorithm service in stage two.

[0172] As can be seen from the above, in the embodiment of the present application, during the video recording process, the camera application executes the first processing process in real time online through the video ISP chip to obtain a basic quality video; after the video recording is completed, the gallery application executes the second processing process offline, which may include video algorithms that have not been executed in the first processing process, and video algorithms that have been executed in the first processing process but whose execution effects did not achieve the optimal target effects due to real-time and other reasons, thereby allowing users to obtain higher quality videos while meeting real-time requirements.

[0173] Based on the above embodiment, the electronic device 200 can enter the video recording process of the general mode, and can also enter the video recording process of the specific video recording mode. Figure 7 Another schematic flow chart of the video processing method provided by the embodiment of the present application is shown, which introduces and explains the video processing process in a specific video recording mode. Figure 7 As shown, the video processing process may include the following steps:

[0174] Step S701: The electronic device 200 displays a video recording interface of a camera application.

[0175] Step S702: The electronic device 200 receives an operation on the video framing interface and enters a target recording mode for video recording.

[0176] It is understandable that the operations acting on the video framing interface include one or more operations for triggering the electronic device 200 to enter the video recording process of the target recording mode.

[0177] The operation acting on the video framing interface may include the operation of the video recording mode control acting on the video framing interface and the operation of the recording control acting on the video framing interface. The operation of the video recording mode control acting on the video framing interface is used to trigger the electronic device 200 to determine that the video recording mode is the target recording mode, and the operation of the recording control acting on the video framing interface is used to trigger the electronic device 200 to enter the video recording of the target recording mode. For example, see Figure 5A The user can select a plurality of video recording modes displayed in the window 59 and then click the video recording control 56 to enter the video recording process of the night scene motion mode.

[0178] The target recording mode refers to a video recording mode different from the normal mode. For example, the target recording mode may include a movie mode, a telephoto mode, a night scene sports mode, and a slow motion mode.

[0179] Step S703: During the video recording process, the electronic device 200 obtains the original image data and metadata required by multiple target video algorithms corresponding to the target recording mode, and performs a first processing process on the original image data through an image signal processor to obtain first video data. The multiple target video algorithms are video algorithms corresponding to the target recording mode.

[0180] It should be noted that for each target recording mode, the target video algorithm to be executed by the gallery application can be preset, that is, the video algorithm to be executed in the second processing process of each target recording mode can be preset. In addition, the execution order of multiple target video algorithms corresponding to the target recording mode can also be preset.

[0181] For example, for the night scene motion mode, video tasks such as noise, image details, motion blur and anti-shake are the key to video quality. Therefore, the target video algorithms corresponding to the night scene motion mode include: deblurring, anti-shake and noise reduction, and the algorithm execution order is: anti-shake, noise reduction, deblurring.

[0182] After entering the video recording process of the target recording mode, the electronic device 200 can collect metadata required by the target video algorithm corresponding to the target recording mode in a targeted manner and store it in a non-volatile storage medium. In this way, when the gallery application further processes the first video file, it can call the metadata of the target video algorithm and execute the target video algorithm based on the metadata.

[0183] It is understandable that some target video algorithms need to use metadata, while some target video algorithms do not. Therefore, when collecting metadata, for target video algorithms that need to use metadata, the metadata required by these target video algorithms are collected; when executing the target video algorithm, for the target video algorithm that needs to use metadata, the metadata pre-collected by the target video algorithm is called to perform video algorithm processing.

[0184] Step S704: If the electronic device 200 receives a second operation on the video framing interface during the video recording process, it exits the video recording, obtains the first video file, stores the first video file and metadata, and updates the first video file to the gallery application. The first video file is a video file generated by encoding the first video data.

[0185] Step S705, the gallery application of the electronic device 200 calls the metadata required by the target video algorithm according to each target video algorithm corresponding to the target recording mode, and executes each target video algorithm in sequence according to the algorithm execution order of each target video algorithm to perform a second processing process on the first video file, obtain a second video file, and store the second video file; the video quality of the second video file is higher than the video quality of the first video file.

[0186] It is understandable that in the second processing, the gallery application sequentially executes multiple target video algorithms on the first video file according to the algorithm execution order. At this time, the second processing includes multiple target video algorithms, and the target video algorithm can be a first type of video algorithm or a second type of video algorithm.

[0187] For example, for the night scene motion mode, the target video algorithms to be executed in the second processing process include: deblurring, anti-shake and noise reduction. Deblurring and anti-shake are the first type of video algorithms, and noise reduction is the second type of video algorithm.

[0188] Optionally, the electronic device 200 may trigger the gallery application to execute the second processing procedure at an appropriate time to call the metadata required by the target video algorithm, execute each target video algorithm in sequence, and obtain a second video file with higher quality.

[0189] The appropriate timing may refer to: when the camera application is detected to be closed or running in the background, the third operation acting on the first thumbnail is received, or the camera application is closed or running in the background and the preset triggering condition is met. For a detailed description, please refer to the corresponding content above, which will not be repeated here.

[0190] It is worth pointing out that the embodiment of the present application is based on a segmented video processing architecture, which can not only enrich the video recording mode, but also make the video obtained in the target recording mode take into account both video quality and real-time requirements, thereby allowing users to obtain videos with better quality and richer video recording modes, thereby improving the user experience.

[0191] Specifically, the segmented video processing architecture may include a first stage and a second stage. The first stage refers to: a process of performing real-time online processing of the Raw image at the video ISP chip end to generate a first video file in real time; the second stage may refer to: a process of processing the first video file at the gallery application end to generate a second video file. For example, see Figure 3 Phase one and phase two.

[0192] Compared with the general video recording mode, a specific video recording mode usually requires the execution of more video algorithms to achieve the video effect of the specific video recording mode. For example, compared with the general recording mode, the night scene motion mode may require the execution of additional anti-shake, deblur and noise reduction algorithms.

[0193] Using a single-stage video processing architecture (e.g. Figure 1B When a specific video recording mode is implemented using the system architecture shown in the figure, the single-stage video processing architecture only has the first stage, and the video ISP chip side needs to process the Raw image in real time online. However, since the specific video recording mode needs to execute more video algorithms, the algorithm processing time of each frame of the Raw image will be longer when the video ISP chip side processes in real time online, and thus the real-time constraints of the video recording cannot be met. In this way, when the video recording ends, the user may not be able to obtain the shot video in real time, and the real-time requirements of the video recording cannot be met. Even when the user ends the video recording, the electronic device 200 may freeze.

[0194] In addition, when using a single-stage video processing architecture to implement a specific video recording mode, if real-time constraints are given priority, the video ISP will sacrifice the algorithm execution effect to meet the requirements of real-time video output when executing the video algorithm, which will lead to poor algorithm execution effect on the video ISP chip side. In this way, although the real-time constraints can be met, the video quality is reduced, resulting in poor video quality for users.

[0195] Therefore, the use of a single-segment video processing architecture usually cannot achieve a variety of video recording modes, and cannot take into account both real-time performance and video quality. In related technologies, in order to prevent users from feeling stuck, the real-time constraints of the video are given priority, sacrificing the algorithm effect of the video algorithm. Although this can achieve a variety of video recording modes and meet the real-time constraints of the video, the quality of the video captured is poor.

[0196] In the embodiment of the present application, a segmented video processing architecture (eg Figure 3 When a specific video recording mode is implemented in the system architecture shown in the figure, the video algorithm executed in the first stage can remain unchanged, and the video algorithm corresponding to the specific video recording mode is put into the second stage for execution. In this way, there is no need to execute more video algorithms in the first stage. When the video ISP chip processes online in real time, the algorithm processing time of each frame of the Raw image can meet the real-time constraint, so that the user can obtain the captured video when the video recording ends, and the captured video obtained in real time will not be too bad; further, the real-time constraint of the second stage is lower, and the video algorithm corresponding to the specific video recording mode is executed in the second stage to implement the specific video recording mode, and a higher quality video can be obtained.

[0197] Therefore, based on the segmented video processing architecture, it is possible to enrich specific video recording modes, and in specific video recording modes, both real-time constraints and video quality can be taken into account.

[0198] Based on the above embodiment, in the second processing process, the gallery application can execute multiple video algorithms to process the first video file to obtain a second video file with higher quality.

[0199] When executing multiple video algorithms, the gallery application can further divide the multiple video algorithms into multiple stages and execute each stage in segments. Of course, it is also possible not to divide into multiple stages. The following will introduce two cases by way of example.

[0200] When the gallery application does not divide the multiple video algorithms into multiple stages for execution, the multiple video algorithms may be executed in sequence to obtain the second video file.

[0201] For example, see Fig. 8A A schematic diagram of the second processing process is shown. After the electronic device 200 obtains the first video file and updates the first video file to the gallery application, the gallery application can first unpack and decode the first video file to obtain the first video data; then execute algorithms 1 to N on the first video data in sequence according to the execution order of multiple video algorithms. Specifically, the first video algorithm is used as the input of algorithm 1 to obtain the execution result of algorithm 1; the execution result of algorithm 1 is used as the input of algorithm 2 to obtain the execution result of algorithm 2; and so on, until the execution of algorithm N is completed and the execution result of algorithm N is obtained; the execution result of algorithm N is encoded and packaged to generate a second video file. The second video file can be viewed by the user.

[0202] Optionally, the gallery application may perform the second processing on the first video file only when the trigger detection module detects a trigger condition. The trigger condition may be: detecting that the camera application is closed or the camera application is running in the background, receiving a third operation acting on the first thumbnail, or the camera application is closed or the camera application is running in the background and a preset trigger condition is met.

[0203] Optionally, if the user actively triggers the gallery application to execute the second processing process, the electronic device 200 can execute multiple video algorithms at one time. User active triggering may refer to user triggering through an interactive control, for example, see Figure 5C, after the mobile phone 51 detects the user's click operation on the trigger control 517, it is considered that the user actively triggered the gallery application to execute the second processing process. If the second processing process is not actively triggered by the user, but the electronic device 200 automatically triggers the second processing process when it determines that the system is idle or the user needs to operate (view, share or edit, etc.) the video, the electronic device 200 can execute multiple video algorithms in the second processing process in stages. When the gallery application divides multiple video algorithms into multiple stages for execution, one stage can include one or more algorithms, and the execution order of the multiple stages is determined by the execution order of the multiple video algorithms.

[0204] For example, in the target recording mode, the target recording mode corresponds to multiple target video algorithms, the execution order of the multiple target video algorithms is preset, and each stage includes a target video algorithm. At this time, the gallery application can, based on the first video file and metadata, execute the staged algorithm processing of each target video algorithm in sequence according to the algorithm execution order to obtain the second video file.

[0205] Among them, the staged algorithm processing process of each target video algorithm can be as follows: detect whether the preset trigger condition is met; if the preset trigger condition is met, decode the first target video file to obtain the first target video data; if the current target video algorithm needs to use metadata, call the metadata required by the target video algorithm, execute the target video algorithm on the first target video data, and obtain the second target video data. If the current target video algorithm does not need to use metadata, the target video algorithm can be executed on the first target video data to obtain the second target video data; encode the second target video data to obtain the second target video file, and store the second target video file; detect whether the preset stop condition is met; if the preset stop condition is not met, enter the staged algorithm processing process of the next target video algorithm; if the preset stop condition is met, determine the second target video file of the current staged algorithm processing process as the second video file; the preset stop condition is to complete the execution of multiple target video algorithms.

[0206] The first target video file is the first video file or the second target video file of the previous staged algorithm processing process, and the first target video data is the video data obtained by decoding the first video data or the second target video file of the previous staged algorithm processing process.

[0207] That is to say, if it is the staged algorithm processing process of the first target video algorithm, the first target video file is the first video file, and the first target video data is the first video data. If it is the staged algorithm processing process of the second and subsequent target video algorithms, the first target video file is the second target video file of the previous staged algorithm processing process.

[0208] Optionally, when the gallery application executes each staged algorithm processing process sequentially, if the current staged algorithm processing process is completed, the second target video file of the previous staged algorithm processing process in the storage medium can be replaced with the second target video file corresponding to the current staged algorithm processing process, that is, the code stream is replaced. At this time, the gallery application can also display thumbnails of the second target video files of each staged algorithm processing process. In other words, after each staged algorithm processing process is completed, the gallery application can display the results of the staged algorithm processing process. In this way, by saving the processing results of each staged algorithm processing process, the user can view the results of each staged algorithm processing process on the gallery application side, thereby improving the user experience.

[0209] Optionally, if during the staged algorithm processing of executing the target video algorithm, the gallery application detects that a preset interruption trigger condition is met, the staged algorithm processing is stopped, and the result data that has been processed in the staged algorithm processing is stored. When it is detected that the preset trigger condition is met, the interrupted staged algorithm processing is continued from the interruption point based on the result data; or, the staged algorithm processing is stopped, the first target video file of the staged algorithm processing is stored, and when it is detected that the preset trigger condition is met, the interrupted staged algorithm processing is re-executed based on the first target video file.

[0210] Optionally, the preset interrupt trigger condition includes at least one of the following: the electronic device 200 is powered off, the device temperature of the electronic device 200 is higher than a preset temperature threshold, the running memory of the electronic device 200 is lower than a preset memory threshold, and receiving an operation for indicating termination of processing, and the operation for indicating termination of processing can be input by the user through an interactive control displayed on the electronic device 200.

[0211] For example, see Figure 8B The schematic diagram of the second processing process being processed in stages is shown. In response to the user's recording operation, the electronic device 200 calls the camera to capture and record the image signal, obtains the Raw image and the metadata required for recording the video algorithm of the subsequent second processing process; the video ISP chip processes the Raw image into video data in real time, and generates a bit stream (i.e., the first video file) according to the video data when the recording ends, stores the bit stream (i.e., the first video file) and metadata, and updates the bit stream to the gallery application, so that the user can view the shot video in the gallery application.

[0212] After the electronic device 200 updates the code stream to the gallery application, the gallery application may perform a second processing process on the code stream to obtain a code stream with higher quality.

[0213] Specifically, the gallery application performs an algorithm trigger detection step to determine whether to perform segmented algorithm processing. When an algorithm trigger condition is detected, segmented algorithm processing is performed; when no algorithm trigger condition is detected, it continues to determine whether an algorithm trigger condition is detected. The algorithm trigger condition may be, for example: a user manually triggers through an interactive button, a user views, edits, or shares a video, the system is idle, and a video is uploaded to a third-party platform.

[0214] When the algorithm trigger condition is detected, the gallery application executes the first algorithm's phased processing. Figure 8B As shown, the staged processing process of the first algorithm includes segmented algorithm 1 data preprocessing and segmented algorithm processing. In the segmented algorithm 1 data preprocessing, the stored bitstream (i.e., the first video file) is unpacked and decoded to obtain the video data, and the metadata required by the current algorithm is parsed from the metadata. After obtaining the video data and the metadata required by the current algorithm based on unpacking decoding and metadata parsing, the video data and the metadata required by the current algorithm are used as algorithm inputs to execute segmented algorithm 1 for video processing to obtain the execution result of segmented algorithm 1. The execution result of segmented algorithm 1 is encoded and packaged to generate a bitstream processed by segmented algorithm 1, and the bitstream in the storage medium is replaced with the bitstream processed by segmented algorithm 1, and the metadata is stored.

[0215] Optionally, after executing the segmented algorithm process of each algorithm, it can also be determined whether a preset stop condition is met, and the preset stop condition is that multiple video algorithms are executed. That is, it is determined whether multiple video algorithms to be executed have been executed. If the preset stop condition is met, the second video file is obtained, and if the preset stop condition is not met, the algorithm trigger detection is continued to be executed to execute the segmented algorithm processing of the next algorithm.

[0216] like Figure 8B As shown, after executing the process of segmented algorithm 1, the gallery application executes the algorithm trigger detection step again. When the algorithm trigger condition is detected, it enters the staged processing process of the second algorithm, that is, executing the data preprocessing of segmented algorithm 2 and the processing of segmented algorithm 2. Similar to the process of segmented algorithm 1, the bitstream in the storage medium is unpacked and decoded to obtain decoded video data; the metadata required by the current algorithm is parsed from the metadata in the storage medium, and based on the decoded video data and the metadata required by the current algorithm, segmented algorithm 2 is executed to perform video processing to obtain the execution result of segmented algorithm 2; the execution result of segmented algorithm 2 is encoded and packaged to generate the bitstream processed by segmented algorithm 2, and the bitstream in the storage medium is replaced with the bitstream processed by segmented algorithm 2, and the metadata is stored.

[0217] Similarly, after the processing of the segmented algorithm N is completed, if it is detected that the preset stop condition is met, the second video file is obtained. At this time, the second video file is the bit stream processed by the segmented algorithm N.

[0218] In comparison, when executing the second processing procedure, the gallery application divides the second processing procedure into multiple stages and executes them in stages, which can further alleviate problems such as device freezing and device heating, and improve user experience. In addition, multiple video algorithms of the second processing procedure are executed in stages, and after each algorithm is executed, it is encoded to generate a video file and stored. In this way, when the algorithm being executed is interrupted, it is not necessary to re-execute all the algorithms, but the interrupted algorithm can be re-executed based on the video file corresponding to the previous video algorithm.

[0219] like Figure 8B As shown, in each segmented algorithm processing process, there is an interrupt processing step. In the interrupt processing step, if it is detected that the preset interrupt trigger condition is met, the interrupt processing process is entered. The preset interrupt trigger condition includes but is not limited to: the user manually terminates through the interactive control, the system is blocked, the electronic device 200 is powered off, and the device temperature of the electronic device 200 is higher than the preset temperature threshold and thermal escape occurs. Figure 8C and Fig.8D The interrupt handling process schematic diagram is shown to introduce and explain the interrupt handling process.

[0220] like Figure 8C As shown, during the operation of the segmented algorithm, interrupt detection is performed in real time to detect whether the preset interrupt trigger condition is met; when it is detected that the preset interrupt trigger condition is not met, the algorithm continues to run; when it is detected that the preset interrupt trigger condition is met, the currently processed part of the video data is saved, and the currently processed part of the video data is encoded and packaged to obtain a partial code stream, and the corresponding code stream in the storage medium is replaced with the partial code stream. When the algorithm triggers the segmented algorithm processing process next time, the algorithm continues to run from the interruption point based on the code stream and metadata stored in the storage medium. That is, when the algorithm is interrupted, the processed results are saved and part of the original code stream is replaced; when the algorithm is triggered to execute again, the algorithm continues to run from the interrupted point until it is completed.

[0221] like Fig.8D As shown, different from Figure 8C, when it is detected that the preset interrupt trigger condition is met, the processed video data is not saved, only the original video data is saved, the original video data is encapsulated and encoded, and a bitstream is generated. The original video data refers to the input of the segmented algorithm. For example, when the segmented algorithm 2 is interrupted, the original video data is the execution result of the segmented algorithm 1. In this case, when the algorithm is interrupted, only the original video data is saved, and when the algorithm is triggered to execute again, the interrupted algorithm is run from the beginning.

[0222] It is worth pointing out that when executing multiple video algorithms, the gallery application usually executes them when the system is idle or the system load is low, the priority of the video algorithm is low, and the user's use of electronic devices will interrupt the execution of the video algorithm. At this time, by setting an interrupt processing mechanism, the video algorithm can be interrupted conveniently, so that the gallery application can better execute the multiple video algorithms of the second processing process.

[0223] As shown above, when the video recording is finished, the electronic device 200 obtains the first video file and updates the first video file to the gallery application. The gallery application can process the first video file again at an appropriate time to obtain a second video file with higher quality.

[0224] After the first video file is updated to the gallery application, a thumbnail of the first video file is displayed in the gallery application. At this time, when the video recording is finished, the thumbnail of the first video file is also displayed in the video framing interface. The user can view the shot video by clicking on the thumbnail of the first video file.

[0225] When the user clicks on the thumbnail of the first video file to play the captured video, but the electronic device 200 has not yet obtained the second video file, in some embodiments, the electronic device 200 can first play the first video file, and trigger the gallery application to process the first video file, obtain the second video file, and replace the first video file in the storage medium with the second video file. In this way, when the user clicks on the thumbnail of the first video file next time, the electronic device 200 can play the second video file. At this time, from the user's perception level, after the captured video has been placed in the gallery application for a period of time, when the user plays the video again, the video picture quality is significantly improved compared to the video picture quality of the first playback.

[0226] In other embodiments, the electronic device 200 may also immediately trigger the gallery application to execute the second processing process, and when the gallery application executes the second processing process, the first video file is not played. At this time, from the user's perception level, after the user clicks the thumbnail of the video file, the electronic device 200 will wait for a period of time before playing the video. During this period of time, the electronic device 200 promptly triggers the gallery application to execute the second processing process on the first video file, obtains the second video file, and plays the second video file after obtaining the second video file.

[0227] In some other embodiments, the electronic device 200 can immediately trigger the gallery application to process the first video file, and send the processed video image to be displayed in real time to perform "playing and processing". Fig. 9 Another schematic flow chart of the video processing method provided by the embodiment of the present application is shown for description. Fig. 9 As shown, the video processing method may include the following steps:

[0228] Step S901: The electronic device 200 displays a video viewing interface of a camera application.

[0229] Step S902: The electronic device 200 receives an operation on the video viewing interface and enters a target recording mode for video recording.

[0230] Step S903: During the video recording process, the electronic device 200 obtains the original image data and metadata required by multiple target video algorithms corresponding to the target recording mode, and performs a first processing process on the original image data through an image signal processor to obtain first video data. The target video algorithm is the video algorithm corresponding to the target recording mode.

[0231] Step S904: If the electronic device 200 receives a second operation on the video framing interface during the video recording process, it exits the video recording, obtains the first video file, stores the first video file and metadata, and updates the first video file to the gallery application. The first video file is a video file generated by encoding the first video data.

[0232] Step S905: The gallery application of the electronic device 200 displays the third thumbnail of the first video file.

[0233] For example, see Figure 5C , a thumbnail 524 is displayed in the video album interface 520, and the thumbnail 524 is the third thumbnail of the first video file.

[0234] Step S906: The electronic device 200 receives a fifth operation applied to the third thumbnail, where the fifth operation is used to trigger the gallery application to play the first video file.

[0235] For example, see Figure 5C , the user can click on the thumbnail 524 in the video album interface 520, and the mobile phone 51 displays the video viewing interface 514 in response to the click operation; the user then clicks on the video location of the video viewing interface 514 to trigger the mobile phone 51 to play the video corresponding to the thumbnail 524. At this time, the fifth operation may include a click operation on the thumbnail 524 and a click operation on the video viewing interface 514.

[0236] Step S907, the gallery application of the electronic device 200 responds to the fifth operation, decodes the first video file, and obtains first video data, where the first video data includes multiple image frames to be processed; for each image frame to be processed, in accordance with the algorithm execution order, the metadata required by the target video algorithm is called, and each target video algorithm is executed on the image frame to be processed in turn to obtain the target image frame, and the target image frame is sent for display.

[0237] In a specific application, after the electronic device 200 obtains the first video data, it caches a preset number of to-be-processed image frames into a task queue, and sequentially performs algorithm processing on each to-be-processed image frame cached in the task queue to obtain a target image frame for each to-be-processed image frame.

[0238] For example, the target recording mode is the night scene motion mode, and the target video algorithm and the algorithm execution order are: anti-shake, noise reduction and deblurring. At this time, for each image frame to be processed, the anti-shake algorithm is first executed on the image frame to be processed to obtain the image frame after anti-shake; it is judged whether all algorithms have been executed. If so, the target image frame is obtained and sent for display; if not, the next video algorithm is continued to be executed; if all video algorithms have not been executed at this time, the noise reduction algorithm is executed on the image frame after anti-shake to obtain the image frame after noise reduction, and it is judged whether all algorithms have been executed; if all algorithms have not been executed at this time, the deblurring algorithm is continued to be executed on the image frame after anti-shake to obtain the image frame after deblurring, and it is judged whether all algorithms have been executed. At this time, if it is determined that all algorithms have been executed, the deblurred image frame is determined as the target image frame, and the target image frame is sent for display; after the current target image frame is sent for display, it is determined whether all image frames have been processed, if so, it is determined that all target image frames have been sent for display, and the encoding and packaging operation is performed on the target image frame to obtain the second video file, if not, the next image frame to be processed is processed. This cycle is repeated until all target image frames to be processed are sent for display.

[0239] Step S908: After each target image frame is sent for display, the electronic device 200 uses the gallery application to encode each target image frame to obtain a second video file, and replaces the first video file in the storage medium with the second video file.

[0240] It is worth noting that when the user needs to play the first video file, the first video file is further processed while the video is being played, and the processed image is displayed in real time to achieve "playing while processing", which improves the user experience. In this way, from the user's perception level, when the user plays the video for the first time, the video can be seen with higher video quality, and the user experience is better.

[0241] In order to better introduce the video processing solution provided by the embodiment of the present application, Figure 5A and Fig.10 The schematic diagram of the video processing process in the night scene motion mode is shown, and the night scene motion mode is used as an example for introduction and explanation.

[0242] like Figure 5A As shown, after the user selects the night scene sports mode option from the multiple video recording modes displayed in the window 59 of the mobile phone 51 and clicks the video recording control 56 on the video framing interface 510, the mobile phone 51 enters the video recording process of the night scene sports mode.

[0243] like Fig.10 As shown, during the video recording process of the night scene motion mode, the mobile phone 51 collects image signals through the camera. At this time, the image sensor of the mobile phone 51 outputs the Raw image and outputs the Raw image to the video ISP chip. In addition, according to the requirements of the night scene motion mode, additional gyro information and Raw images are collected as metadata. Optionally, in order to save storage space, a Raw image with a low frame rate (1-5fps) can be collected.

[0244] The video ISP chip processes the Raw image into a full field of view (FOV) video, and when the video recording is finished, stores the full FOV raw bitstream and metadata in the read-only memory (ROM) of the mobile phone 51. Metadata includes the Raw image and Gyro information. For example, the metadata in the ROM can be as follows: Fig.11 As shown, image information such as Raw images and depth maps can be stored in the Trak of the MP4 file, while non-image information such as Gyro can be stored in the metadata of the MP4 file.

[0245] After the original bitstream of the full FOV is stored in the ROM, the user can view the original bitstream of the full FOV through the gallery application. At this point, the processing of stage 1 is completed.

[0246] The gallery application of the mobile phone 51 performs algorithm trigger detection in real time to determine whether an algorithm trigger condition is detected; if an algorithm trigger condition is detected, the following algorithms are executed serially in sequence: anti-shake algorithm, noise reduction algorithm and deblurring algorithm.

[0247] When the anti-shake algorithm is triggered, the gallery application first pre-processes the anti-shake data. Specifically, the original bitstream of the full FOV is read from the ROM, and the Gyro information is parsed from the metadata. Then, based on the original bitstream of the full FOV and the Gyro information, the anti-shake algorithm is processed to obtain the anti-shake video. Finally, the original bitstream of the full FOV in the ROM is replaced with the anti-shake video, and the metadata continues to be stored. At this point, the user sees the anti-shake video in the gallery application.

[0248] When the noise reduction algorithm is triggered, the gallery application first performs noise reduction algorithm preprocessing, that is, decodes the stabilized video to obtain the decoded video, and parses the low fps Raw image from the metadata. Then, based on the decoded video and Raw image, the noise reduction algorithm is executed to obtain the noise-reduced video. Finally, the stabilized video in the ROM is replaced with the noise-reduced video, and the metadata continues to be stored. At this point, the user sees the stabilized and noise-reduced video in the gallery application.

[0249] It is worth noting that the Raw image includes a large amount of image details. Through the Raw image, a large amount of image details lost due to noise removal when the video ISP executes the noise reduction algorithm can be restored, thereby improving the video quality.

[0250] When the deblurring algorithm is triggered in the gallery application, the deblurring algorithm is preprocessed first, that is, the denoised video is decoded to obtain the decoded video, and the Gyro information is parsed from the metadata. Then, based on the decoded video and Gyro information, the deblurring algorithm is executed to obtain the deblurred video. Finally, the stabilized video in the ROM is replaced with the deblurred video. At this time, the metadata can continue to be stored or deleted. In this way, the user can view the stabilized, denoised and deblurred video in the gallery application.

[0251] It is understandable that there will be interruption detection processing during the operation of the anti-shake algorithm, noise reduction algorithm, and deblurring algorithm. Please refer to the above for specific introduction and will not be repeated here.

[0252] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0253] In the embodiments provided in the present application, it should be understood that the disclosed devices, electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0254] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0255] The electronic device provided in the embodiments of the present application may include a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, a method as described in any one of the above method embodiments is implemented.

[0256] The embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0257] An embodiment of the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0258] The present application also provides a chip system, which includes a processor coupled to a memory, and the processor executes a computer program stored in the memory to implement the methods described in the above method embodiments. The chip system can be a single chip or a chip module composed of multiple chips.

[0259] In the above-mentioned embodiments, the description of each embodiment has its own emphasis. For the part that is not described or recorded in detail in a certain embodiment, please refer to the relevant description of other embodiments. It should be understood that the size of the sequence number of each step in the above-mentioned embodiment does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. In addition, in the description of the present application specification and the attached claims, the terms "first", "second", "third", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance. The reference "one embodiment" or "some embodiments" described in the present application specification means that one or more embodiments of the present application include specific features, structures or characteristics described in combination with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in the different places in this specification are not necessarily all referring to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways.

[0260] Finally, it should be noted that the above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A video processing method, characterized in that: Applied to an electronic device, the electronic device is installed with a camera application and a gallery application; the method comprises: Displaying the video framing interface of the camera application; receiving a first operation on the video framing interface and entering video recording; During the video recording process, original image data and metadata required by a video algorithm of a second processing process are obtained, and a first processing process is performed on the original image data by an image signal processor to obtain first video data; If a second operation acting on the video framing interface is received during the video recording process, the video recording is exited, a first video file is obtained, the first video file and the metadata are stored, and the first video file is updated to the gallery application, where the first video file is a video file generated by encoding the first video data; The gallery application calls the metadata to perform a second processing process on the first video file according to the video algorithm of the second processing process, obtains a second video file, and stores the second video file; the video quality of the second video file is higher than the video quality of the first video file; Among them, the second processing process includes a first type of video algorithm and / or a second type of video algorithm, the first type of video algorithm is a video algorithm that has not been executed in the first processing process, and the second type of video algorithm is a video algorithm that has been executed in the first processing process but the execution effect does not achieve the optimal target effect.

2. The method according to claim 1, characterized in that The gallery application calls the metadata to perform a second processing process on the first video file according to the video algorithm of the second processing process to obtain a second video file, including: If the camera application is closed, running in the background, or receives a third operation on the first thumbnail, the gallery application calls the metadata to perform a second processing process on the first video file according to the video algorithm of the second processing process to obtain a second video file, and the first thumbnail is the thumbnail of the first video file displayed on the video recording interface after receiving the second operation.

3. The method according to claim 2, characterized in that If the camera application is closed or the camera application is running in the background, the gallery application calls the metadata to perform a second processing process on the first video file according to the video algorithm of the second processing process to obtain a second video file, including: If the camera application is closed or is running in the background and a preset trigger condition is detected, the gallery application calls the metadata to perform a second processing process on the first video file according to the video algorithm of the second processing process to obtain the second video file; The preset trigger condition includes any one of the following: receiving a fourth operation acting on the second thumbnail or the trigger control, an operation of opening the gallery application, an operation of viewing or playing the first video file, an operation of sharing the first video file, an operation of editing the first video file, the system of the electronic device is in an idle state, and the screen of the electronic device is turned off or off; The second thumbnail is a thumbnail of the first video file displayed in the gallery application; the trigger control is a control displayed in the gallery application for triggering video algorithm processing of the first video file.

4. The method according to any one of claims 1 to 3, characterized in that: The receiving a first operation on the video framing interface and entering video recording includes: receiving an operation of a video recording mode control on the video framing interface, and determining that the video recording mode is a target recording mode; An operation of a recording control acting on the video framing interface is received to enter video recording of the target recording mode; wherein the first operation includes the operation acting on the video recording mode control and the operation acting on the recording control.

5. The method according to claim 4, characterized in that The gallery application calls the metadata to perform a second processing process on the first video file according to the video algorithm of the second processing process to obtain the second video file, including: The gallery application sequentially executes the phased algorithm processing of each target video algorithm in accordance with the algorithm execution order of the multiple target video algorithms corresponding to the target recording mode based on the first video file and the metadata to obtain the second video file; the video algorithm of the second processing process includes the multiple target video algorithms, and the target video algorithm is the first type of video algorithm or the second type of video algorithm; The phased algorithm processing process of each target video algorithm includes: Detecting whether the preset trigger condition is met; If the preset trigger condition is met, a decoding operation is performed on the first target video file to obtain first target video data; the target video algorithm is executed on the first target video data to obtain second target video data, or the metadata required by the target video algorithm is called to execute the target video algorithm on the first target video data to obtain second target video data; an encoding operation is performed on the second target video data to obtain a second target video file, and the second target video file is stored; Detect whether the preset stop condition is met; If the preset stop condition is not met, then entering the next staged algorithm processing process of the target video algorithm; If the preset stop condition is met, determining the second target video file of the current staged algorithm processing process as the second video file; The preset stop condition is that the plurality of target video algorithms are executed; The first target video file is the first video file or the second target video file of the previous staged algorithm processing process, and the first target video data is the video data obtained by decoding the first video data or the second target video file of the previous staged algorithm processing process.

6. The method according to claim 5, characterized in that The method further comprises: If during the staged algorithm processing of executing the target video algorithm, it is detected that a preset interrupt trigger condition is met; Stop executing the staged algorithm processing process, and store the result data that has been processed in the staged algorithm processing process, and when it is detected that the preset trigger condition is met, continue to execute the interrupted staged algorithm processing process from the interruption point based on the result data; or stop executing the staged algorithm processing process, store the first target video file of the staged algorithm processing process, and when it is detected that the preset trigger condition is met, re-execute the interrupted staged algorithm processing process based on the first target video file.

7. The method according to claim 6, characterized in that The preset interrupt trigger condition includes at least one of the following: the electronic device is powered off, the device temperature of the electronic device is higher than a preset temperature threshold, the running memory of the electronic device is lower than a preset memory threshold, and receiving an operation for indicating termination of processing.

8. The method according to claim 5, characterized in that The method further comprises: When executing each of the staged algorithm processing processes in sequence, if the current staged algorithm processing process is completed, the second target video file of the previous staged algorithm processing process in the storage medium is replaced with the second target video file corresponding to the current staged algorithm processing process; The gallery application displays thumbnails of the second target video files during each phase of the algorithm processing.

9. The method according to claim 4, characterized in that The method further comprises: The gallery application displays a third thumbnail of the first video file; The gallery application calls the metadata to perform a second processing process on the first video file according to the video algorithm of the second processing process to obtain the second video file, including: receiving a fifth operation applied to the third thumbnail, wherein the fifth operation is used to trigger the gallery application to play the first video file; In response to the fifth operation, the gallery application decodes the first video file to obtain first video data, where the first video data includes a plurality of image frames to be processed; for each of the image frames to be processed, in accordance with the algorithm execution order, the metadata required by the target video algorithm is called, each of the target video algorithms is sequentially executed on the image frames to be processed, a target image frame is obtained, and the target image frame is sent for display; After each of the target image frames is displayed, the gallery application performs encoding operations on each of the target image frames to obtain the second video file, and replaces the first video file in the storage medium with the second video file.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 9 is implemented.

11. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images