Video processing method, electronic device, and computer-readable storage medium

Abstract

Embodiments of the present application disclose a video processing method, an electronic device, and a computer-readable storage medium. The method comprises: an electronic device obtaining raw image data and metadata in a video recording process, and processing the raw image data by means of an image signal processor to obtain first video data; when stopping video recording, obtaining a first video file, and storing the first video file and the metadata, the first video file being a video file generated by encoding the first video data; and on the basis of a video algorithm that needs to be executed, a gallery application calling required metadata to process the first video file so as to obtain a second video file, the video quality of the second video file being higher than the video quality of the first video file. In this way, a basic quality video is obtained at the end of the video recording, so that real-time requirements are met; and the basic quality video is further processed in the gallery application, so that a user obtains a higher quality video while the real-time requirements are met.

Description

Video processing method, electronic device, and computer-readable storage medium

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on September 18, 2023, with application number 202311209386.2 and application name “Video Processing Method, Electronic Device and Computer-Readable Storage Medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of image technology, and in particular to a video processing method, an electronic device, and a computer-readable storage medium. Background Art

[0003] With the increasing popularity of electronic devices such as mobile phones, video recording using the video recording function of mobile phones and other electronic devices has gradually become a trend.

[0004] Currently, when recording videos, mobile phones and other electronic devices first capture the original image (Raw image) through a camera, and then use an image signal processor (ISP) to process the Raw image frame by frame in real time to obtain the processed video image. At the end of the video recording, mobile phones and other electronic devices can obtain the final video generated by multiple frames of video images. This final video is the video captured by the user.

[0005] However, the processing time of the ISP on the raw image is often subject to real-time constraints, resulting in lower quality of the video image processed by the ISP, and thus lower quality of the video obtained by the user.

[0006] Summary of the Invention

[0007] The embodiments of the present application provide a video processing method, an electronic device, and a computer-readable storage medium, which can solve the problem of poor video quality of videos captured by existing electronic devices.

[0008] In a first aspect, an embodiment of the present application provides a video processing method, which is applied to an electronic device, wherein the electronic device is installed with a camera application and a gallery application. After the electronic device displays the video framing interface of the camera application, it receives a first operation applied to the video framing interface and enters video recording; during the video recording process, it obtains the original image data and metadata required by the video algorithm of the second processing process, and performs the first processing process on the original image data through the image signal processor to obtain first video data; if the second operation applied to the video framing interface is received during the video recording process, the video recording is exited, a first video file is obtained, and the first video file and metadata are stored, and the first video file is updated to the gallery application, where the first video file is a video file generated by encoding the first video data; the gallery application calls the metadata according to the video algorithm of the second processing process to perform a second processing process on the first video file to obtain a second video file, and stores the second video file; the video quality of the second video file is higher than that of the first video file.

[0009] Among them, the second processing process includes the first type of video algorithm and / or the second type of video algorithm. The first type of video algorithm is a video algorithm that has not been executed in the first processing process, and the second type of video algorithm is a video algorithm that has been executed in the first processing process but the execution effect has not achieved the optimal target effect.

[0010] As can be seen from the above, during the video recording process, the embodiment of the present application pre-collects the metadata required for the second processing process to be performed by the subsequent gallery application, and performs the first processing process on the original image data in real time through the ISP, so that the first video file can be obtained at the end of the video recording. The first video file can be viewed by the user, meeting the real-time requirements of the video recording; after obtaining the first video file, the metadata is called on the gallery application side to further perform the second processing process on the first video file to obtain a second video file with higher quality. In this way, a basic quality video (i.e., the first video file) can be obtained at the end of the video recording, meeting the real-time requirements. After the first video file is updated to the gallery application, the gallery application side further executes the video algorithms that have not been executed in the first processing process based on the pre-collected metadata, as well as the video algorithms that have been executed in the first processing process but have not achieved the optimal target effect due to real-time reasons, so as to improve the video quality of the basic quality video obtained by shooting and obtain a higher quality video. Therefore, under the premise of meeting the real-time requirements, the user can obtain a higher quality video, thereby improving the user experience.

[0011] In some possible implementations of the first aspect, if the electronic device detects that the camera application is closed or running in the background, or receives a third operation on the first thumbnail, the gallery application invokes the metadata to perform the second processing on the first video file according to the video algorithm of the second processing process to obtain a second video file. The first thumbnail is the thumbnail of the first video file displayed on the video viewfinder interface after receiving the second operation, that is, the thumbnail of the first video file is displayed on the video viewfinder interface when exiting video recording.

[0012] In comparison, when video recording is finished but the camera is running in the foreground, if the gallery application calls the metadata to perform the second processing on the first video file, since the camera application still needs to occupy computing resources and memory resources to perform image acquisition and preview video streams and other processing processes, and the gallery application also needs to occupy computing resources and memory resources to run the video algorithm of the second processing process, a large amount of computing resources and memory resources may be occupied, and the processor may be in a high-load state for a long time, which may lead to problems such as device freezing, device heating and high device power consumption, affecting the user experience.

[0013] After the video recording is completed, the camera application is closed, the camera application is running in the background, or a third operation is received on the first thumbnail, the gallery application calls the metadata to perform the second processing on the first video file. Since the electronic device does not need to perform image acquisition and preview video streams at this time, the probability of problems such as device freeze, device overheating, and high device power consumption is low, thereby improving the user experience.

[0014] When video recording is finished and the camera app is in the foreground, if the user needs to view the captured video (i.e., receives the third operation), the electronic device can promptly trigger the gallery app to further process the basic quality video based on the metadata and promptly obtain a higher quality second video file, allowing the user to view the higher quality video in a timely manner. When the user views the first video file, the camera app jumps to the background and enters the video preview state of the gallery app. At this time, there is no need to perform image acquisition and preview video stream processing.

[0015] In some possible implementations of the first aspect, if the camera application is closed or running in the background and a preset trigger condition is detected, the gallery application invokes the metadata to perform the second processing on the first video file according to the video algorithm of the second processing process to obtain a second video file;

[0016] The preset trigger condition includes any one of the following: receiving a fourth operation acting on the second thumbnail or the trigger control, an operation of opening a gallery application, an operation of viewing or playing the first video file, an operation of sharing the first video file, an operation of editing the first video file, the system of the electronic device being in an idle state, and the screen of the electronic device being turned off or on;

[0017] The second thumbnail is a thumbnail of the first video file displayed in the gallery application; the trigger control is a control displayed in the gallery application for triggering video algorithm processing of the first video file.

[0018] In this implementation, when the camera application is running in the background or closed, it further determines whether the preset trigger conditions are met. When the preset trigger conditions are met, the gallery application calls the metadata to further process the first video file. In this way, when the user needs to view, edit or share the captured video, the gallery application can process the first video file in time, so that the user can obtain higher quality video in time; in addition, the gallery application can also call the metadata to further process the first video file when the system is idle, the screen is off or the screen is turned off, which can further alleviate problems such as device freezes and device heating.

[0019] In some possible implementations of the first aspect, receiving a first operation on a video framing interface and entering video recording includes:

[0020] Receive an operation on a video recording mode control on a video framing interface, and determine that the video recording mode is a target recording mode; receive an operation on a recording control on a video framing interface, and enter video recording in the target recording mode; wherein the first operation includes an operation on a video recording mode control and an operation on a recording control.

[0021] In this case, the video algorithms of the second processing step include multiple target video algorithms corresponding to the target recording mode. Therefore, when collecting metadata, metadata required by the target video algorithms can be collected. Among the multiple target video algorithms, some require metadata for execution, while others do not.

[0022] In the target recording mode scenario, the multiple target video algorithms corresponding to the target recording mode have a preset algorithm execution order. At this time, the gallery application executes each target video algorithm in sequence according to the algorithm execution order of the multiple target video algorithms. When executing the multiple target video algorithms, if the target video algorithm requires metadata, the corresponding metadata is called for video algorithm processing. If the target video algorithm does not require metadata, the metadata can be not called for video algorithm processing. By calling the metadata required by the target video algorithm, executing the multiple target video algorithms in sequence, and then performing the encoding operation, a second video file can be obtained; at this time, the video algorithm of the second processing process includes the multiple target video algorithms corresponding to the target recording mode, and the target video algorithm can be a first-category video algorithm or a second-category video algorithm.

[0023] In this implementation, a video processing framework based on executing a first processing process during video recording and a second processing process in a gallery application after video recording is completed can enrich the video recording mode and can take into account both video quality and real-time requirements in the target recording mode, so that users can obtain videos with better quality and richer video recording modes, thereby improving user experience.

[0024] In some possible implementations of the first aspect, the gallery application, based on a video algorithm of the second processing process, calls metadata to perform the second processing process on the first video file to obtain a second video file, including:

[0025] The gallery application sequentially executes the phased algorithm processing of each target video algorithm based on the first video file and metadata in the algorithm execution order of the multiple target video algorithms corresponding to the target recording mode to obtain a second video file;

[0026] The phased algorithm processing of each target video algorithm includes:

[0027] detecting whether a preset trigger condition is satisfied; if the preset trigger condition is satisfied, performing a decoding operation on the first target video file to obtain first target video data; if the current target video algorithm does not require metadata, executing the target video algorithm on the first target video data to obtain second target video data; and if the current target video algorithm requires metadata, calling the metadata required by the current target video algorithm, executing the target video algorithm on the first target video data to obtain second target video data; performing an encoding operation on the second target video data to obtain a second target video file, and storing the second target video file;

[0028] Detecting whether a preset stop condition is met; if the preset stop condition is not met, entering the staged algorithm processing process of the next target video algorithm; if the preset stop condition is met, determining the second target video file of the current staged algorithm processing process as the second video file; the preset stop condition is that multiple target video algorithms have been executed;

[0029] The first target video file is the first video file or the second target video file of the previous staged algorithm processing process, and the first target video data is the video data obtained by decoding the first video data or the second target video file of the previous staged algorithm processing process.

[0030] In this implementation, by executing the second processing procedure in stages, problems such as device freezing and device heating can be further alleviated, thereby improving user experience.

[0031] In some possible implementations of the first aspect, the method further includes: if a preset interrupt trigger condition is detected during the execution of the staged algorithm processing of the target video algorithm; stopping the staged algorithm processing and storing the result data of the staged algorithm processing, and when the preset trigger condition is detected to be met, resuming the interrupted staged algorithm processing from the interruption point based on the result data; or stopping the staged algorithm processing, storing the first target video file of the staged algorithm processing, and re-executing the interrupted staged algorithm processing based on the first target video file when the preset trigger condition is detected to be met. In some possible implementations of the first aspect, the preset interrupt trigger condition includes at least one of the following: the electronic device is powered off, the device temperature of the electronic device is higher than a preset temperature threshold, the running memory of the electronic device is lower than a preset memory threshold, and receiving an operation for instructing to terminate the processing. In this way, by executing the algorithm processing of each target video algorithm in stages and setting an interrupt mechanism, relevant data can be saved in a timely manner when the algorithm processing is interrupted, without having to re-execute all the target video algorithms, thereby better implementing the second processing process.

[0032] In some possible implementations of the first aspect, the method also includes: when executing each staged algorithm processing process sequentially, if the current staged algorithm processing process is completed, replacing the second target video file of the previous staged algorithm processing process in the storage medium with the second target video file corresponding to the current staged algorithm processing process; the gallery application displays thumbnails of the second target video files of each staged algorithm processing process.

[0033] In this implementation, by saving the processing results of each staged algorithm processing process, the user can view the results of each staged algorithm processing process on the gallery application side, thereby improving the user experience.

[0034] In some possible implementations of the first aspect, the method further includes:

[0035] The gallery application displays a third thumbnail of the first video file;

[0036] The gallery application performs the second processing on the first video file by calling the metadata according to the video algorithm of the second processing process to obtain the second video file, including:

[0037] receiving a fifth operation applied to the third thumbnail, the fifth operation being used to trigger the gallery application to play the first video file;

[0038] In response to the fifth operation, the gallery application decodes the first video file to obtain first video data, which includes multiple image frames to be processed; for each image frame to be processed, according to the algorithm execution order, the metadata required by the target video algorithm is called, and each target video algorithm is executed on the image frame to be processed in turn to obtain the target image frame, and the target image frame is sent for display; wherein, in the process of sequentially executing each target video algorithm, if a target video algorithm does not require metadata, the metadata is not used for video algorithm processing; if a target video algorithm requires metadata, the corresponding metadata is called for video algorithm processing.

[0039] After each target image frame is sent for display, the gallery application performs encoding operations on each target image frame to obtain a second video file, and replaces the first video file in the storage medium with the second video file.

[0040] In this implementation, when a user needs to play a first video file, the first video file is further processed while the video is playing, and the processed image is displayed in real time to achieve "playing while processing", thereby improving the user experience.

[0041] In a second aspect, an embodiment of the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in any one of the first aspects above when executing the computer program.

[0042] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the method of any one of the first aspects above.

[0043] In a fourth aspect, embodiments of the present application provide a chip system, comprising a processor coupled to a memory, the processor executing a computer program stored in the memory to implement any of the methods described in the first aspect. The chip system may be a single chip or a chip module consisting of multiple chips.

[0044] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when executed on an electronic device, enables the electronic device to execute the method described in the first aspect above.

[0045] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] FIG1A is a schematic diagram of a video recording scenario provided in an embodiment of the present application;

[0047] FIG1B is a schematic diagram of a video processing process provided by an embodiment of the present application;

[0048] FIG1C is a schematic diagram of the ISP processing process provided in an embodiment of the present application;

[0049] FIG1D is a schematic diagram of the ISP chip processing process provided by an embodiment of the present application;

[0050] FIG2 is a schematic structural diagram of an electronic device 200 provided in an embodiment of the present application;

[0051] FIG3 is a schematic block diagram of the video recording system architecture of the electronic device 200 provided in an embodiment of the present application;

[0052] FIG4 is a schematic block diagram of a process of a video processing method provided by an embodiment of the present application;

[0053] FIG5A is a schematic diagram of a video recording process provided in an embodiment of the present application;

[0054] FIG5B is a schematic diagram of a video recording process provided in an embodiment of the present application;

[0055] FIG5C is a schematic diagram of video operation provided by an embodiment of the present application;

[0056] FIG6 is a schematic block diagram of another process of a video processing method provided in an embodiment of the present application;

[0057] FIG7 is a schematic block diagram of another process of the video processing method provided in an embodiment of the present application;

[0058] FIG8A is a schematic diagram of a second processing process provided in an embodiment of the present application;

[0059] FIG8B is a schematic diagram of the second processing step provided by an embodiment of the present application;

[0060] FIG8C is a schematic diagram of an interrupt handling process provided by an embodiment of the present application;

[0061] FIG8D is a schematic diagram of an interrupt handling process provided by an embodiment of the present application;

[0062] FIG9 is a schematic block diagram of another process of the video processing method provided in an embodiment of the present application;

[0063] FIG10 is a schematic diagram of a video processing process in a night scene motion mode according to an embodiment of the present application;

[0064] FIG11 is a schematic diagram of metadata storage provided in an embodiment of the present application. DETAILED DESCRIPTION

[0065] In the following description, for the purpose of illustration rather than limitation, specific details such as particular system structures and technologies are provided to facilitate a thorough understanding of the embodiments of the present application.

[0066] 1. Video recording (video recording) scene.

[0067] The following is an exemplary introduction to the video recording process with reference to Figures 1A and 1B , wherein Figure 1A is a schematic diagram of a video recording scenario provided by an embodiment of the present application, and Figure 1B is a schematic block diagram of a video recording system architecture provided by an embodiment of the present application.

[0068] As shown in FIG1A , the main interface 111 of the mobile phone 11 displays icons such as a camera application 112, a gallery application 113, a calendar application, and a settings application. Upon receiving a click operation on the icon of the camera application 112, the mobile phone 111 displays a video preview interface 114 of the camera application 112 in response to the click operation. The video preview interface 114 includes a viewfinder and controls 115, and the viewfinder displays the image captured by the camera of the mobile phone 11.

[0069] After receiving the click operation on the control 115 , the mobile phone 11 enters the video recording mode in response to the click operation and displays the video recording interface 116 . The video recording interface 116 includes the video being recorded and the control 117 .

[0070] After receiving the click operation on the control 117, the mobile phone 11 exits the video recording in response to the click operation, obtains the recorded video, displays a thumbnail 118 of the recorded video on the video preview interface, and updates the recorded video to the gallery application 113. The user can jump to the gallery application 113 to view the recorded video by clicking the thumbnail 118, or enter the gallery application 113 to view the recorded video by clicking the icon of the gallery application 113.

[0071] As shown in Figure 1B, the recording system framework includes: an application layer, an intermediate layer and a functional module layer. The application layer includes a camera application and a gallery application. The intermediate layer includes a camera service and a media library. The camera service includes a real-time recording sub-service. The real-time recording sub-service includes camera devices and camera streams, etc. The real-time recording sub-service is used to implement the recording function. The media library includes a Media Store database interface and a Media Provider-DB (Media Provider database). The functional module layer includes an acquisition module, an algorithm processing module and a file system, etc. The acquisition module is used to acquire image data. The algorithm processing module is used to execute multiple algorithm processes to perform real-time online processing of image data. The file system is used to store recorded videos, which are usually stored in non-volatile storage media, such as read-only memory (ROM).

[0072] In the scenario shown in Figure 1A, after mobile phone 11 receives a click on control 115, the camera application initiates a recording request to the camera service. In response to the recording request, the camera service invokes the acquisition module to capture images, obtains the image data to be processed, and stores the processed image data in the image buffer. The algorithm processing module then performs algorithmic processing on the processed image data in real time. During the algorithm processing phase, multiple algorithmic processes (procs) are sequentially executed on the processed image data to obtain a real-time video stream, which is then returned to the camera application via the camera service.

[0073] After receiving a click operation on control 117, the video encoding module of mobile phone 11 performs operations such as encapsulation and encoding on the video image output by the algorithm processing module to obtain the recorded video when video recording ends, and updates the recorded video to the gallery application. The gallery application can access the recorded video in the file system through the media library. Figure 1B exemplifies that the recorded video is a 100-point basic quality video.

[0074] 2. Video processing process.

[0075] In a video recording scenario, the electronic device uses the ISP chip to perform real-time online processing on the images captured by the camera, so that the recorded video can be obtained when the video recording ends, ensuring the real-time nature of the video recording. The processing process of the ISP chip may include the process of the algorithm processing module in Figure 1B.

[0076] For example, referring to FIG1C , which shows a schematic diagram of the video processing process provided by an embodiment of the present application, based on the video recording scene shown in FIG1A , the camera devices such as the lens and image sensor of the mobile phone 11 capture the light signal of the framing environment, convert the light signal into an electrical signal, and output a Raw image (i.e., original image data); the ISP chip processes each frame of the Raw image in real time online by executing multiple processing algorithms in real time to obtain a processed image; the processed image output by the ISP chip can be divided into two data streams, one of which is a preview video stream and the other is a recording video stream. The preview video stream refers to the video stream displayed on the display screen of the electronic device during the video recording process, and the recording video stream refers to the video stream stored in the electronic device after the video recording is completed and available for the user to view. As shown in Figure 1B, after the encoding module performs video encoding on the processed image output by the ISP chip, it obtains a preview video stream and sends the preview video stream for display to display the video screen on the display screen, for example, the video screen displayed on the video preview interface 114 and the video recording interface 116 in Figure 1A; in addition, another encoding module performs video encoding and other operations on the processed image output by the ISP chip to generate a video file, stores the video file in a storage medium, and updates the video file to the gallery application 113, so that the user can view the video file in the gallery application 113.

[0077] In the related art, after a video file is updated to the gallery application 113, the gallery application 113 only supports viewing, editing, and sharing of the video file, and does not perform further processing on the video file. After the captured video is stored in the gallery application 113 for a period of time, the video quality does not change.

[0078] The ISP chip executes multiple processing algorithms in a pipelined manner to enhance image quality, such as detail, color, and brightness. For example, referring to FIG1D , which illustrates the schematic diagram of the ISP chip processing process provided by an embodiment of the present application, a Raw image is input to the ISP and subsequently processed through algorithms such as bad pixel correction, black level correction, noise reduction, lens brightness correction, automatic white balance, color interpolation, color correction, Ygamma, sharpening, and color space conversion to obtain a YUV image. The YUV image undergoes operations such as video encoding and packaging to produce a video file.

[0079] It is understandable that the ISP chip may include more or fewer processing algorithms than those in FIG1D . For example, the ISP chip may also include an anti-shake algorithm and a beauty algorithm.

[0080] During the long-term research process, the inventors found that the video recording in the related art cannot take into account both real-time performance and video quality at the same time, and usually sacrifices video quality to ensure the real-time performance of video recording.

[0081] Specifically, to ensure real-time video recording so that users can access the recorded video as soon as recording ends, the processing time for each raw image frame is often subject to real-time constraints. This forces the ISP to run only basic processing algorithms to process the raw images, sacrificing the effectiveness of each processing algorithm to meet the real-time video output requirement. This way, while the real-time requirements of video recording can be met, the resulting video file has poor quality. For example, in the scenario shown in FIG1A , when the user clicks control 117 to end video recording, the mobile phone 11 needs to generate a video file in real time for the user to view. In this case, when the user records a 30fps (Frame Per Second) video, the processing time for each raw image frame may be around 33 milliseconds. Within 33 milliseconds, the number and types of image processing algorithms that the ISP can execute are limited, and the effectiveness of some image processing algorithms is also sacrificed, resulting in a very limited execution of the image processing algorithms, which in turn results in poor video quality in the resulting video file.

[0082] In response to the related issues mentioned above, an embodiment of the present application provides a video processing solution, which processes the original image data in real time on the ISP chip side during the video recording process, so as to obtain a first video file in real time when the video recording ends, thereby meeting the real-time requirements of video recording; after the video recording ends, the gallery application side further processes the first video file based on metadata to obtain a second video file with higher quality.

[0083] Different from executing the processing algorithm serially in real time online only through the ISP chip during the video recording process, the video processing solution of the embodiment of the present application not only executes the processing algorithm serially in real time online through the ISP chip during the video recording process, but also performs further offline processing on the captured video file based on metadata at the gallery application end after the video recording is completed to obtain a higher quality video file. In this way, a basic quality video (i.e., the first video file) can be obtained at the end of the video recording, meeting the real-time requirements of the video recording; after the video recording is completed, the basic quality video obtained by the capture is further processed offline based on the pre-collected metadata at the gallery application end to obtain a higher quality video. Therefore, under the premise of meeting the real-time requirements, users can obtain higher quality videos, which improves the user experience.

[0084] The video processing solution provided in the embodiments of the present application can be applied to an electronic device, which may be, for example, a mobile phone or a tablet computer, and the type of electronic device is not limited here.

[0085] For example, referring to the structural diagram of the electronic device 200 shown in FIG2 , the electronic device 200 may include but is not limited to: a processor 210 , a memory 220 , a camera 230 , and a display screen 240 .

[0086] It is understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 200. In other embodiments of the present application, the electronic device 200 may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The illustrated components can be implemented in hardware, software, or a combination of software and hardware. For example, when the electronic device 200 is a mobile phone, the electronic device 200 may further include: a universal serial bus (USB) interface, a charging management module, a sensor module, a power management module, a battery, an antenna, a mobile communication module, a wireless communication module, an audio module, a speaker, a receiver, a microphone, a headphone jack, a button, a motor, an indicator, and a subscriber identification module (SIM) card interface; the sensor module may include at least one of the following: a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a temperature sensor, a touch sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, an ambient light sensor, and a bone conduction sensor, etc.

[0087] The processor 210 may include one or more processing units, for example: the processor 210 may include an application processor (AP), a graphics processing unit (GPU), an ISP, a controller, a video codec, a digital signal processor (DSP), etc. Among them, different processing units can be independent devices or integrated into one or more processors. The controller can generate an operation control signal based on the instruction opcode and the timing signal to complete the control of instruction fetching and execution. It can be understood that when the gallery application of the embodiment of the present application calls the metadata to further process the first video file, one or more processors such as AP, GPU, ISP and DSP may be involved. At this time, the ISP is processed offline.

[0088] In some embodiments, the processor 210 may include one or more interfaces, such as an inter-integrated circuit (I2C) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, and the like.

[0089] The I2C interface is a bidirectional synchronous serial bus that includes a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 210 may include multiple I2C busses. The processor 210 may be coupled to the camera 230, etc., through different I2C bus interfaces.

[0090] The MIPI interface can be used to connect the processor 210 to peripheral devices such as the display 240 and the camera 230. MIPI interfaces include the camera serial interface (CSI) and the display serial interface (DSI). In some embodiments, the processor 210 and the camera 230 communicate via the CSI interface to implement the camera function of the electronic device 200. The processor 210 and the display 240 communicate via the DSI interface to implement the display function of the electronic device 200.

[0091] The GPIO interface can be configured through software. The GPIO interface can be configured as a control signal or a data signal. In some embodiments, the GPIO interface can be used to connect the processor 210 to the camera 230, the display 240, etc. The GPIO interface can also be configured as an I2C interface, a MIPI interface, etc.

[0092] It is understood that the interface connection relationship between the modules illustrated in the embodiment of the present application is merely an illustrative illustration and does not constitute a structural limitation on the electronic device 200. In other embodiments of the present application, the electronic device 200 may also adopt a different interface connection method from the above embodiment, or a combination of multiple interface connection methods.

[0093] Electronic device 200 implements display functionality through a GPU, display screen 240, and an application processor. The GPU is a microprocessor for image processing that connects display screen 240 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 210 may include one or more GPUs that execute program instructions to generate or modify display information.

[0094] Display screen 240 is used to display images and videos, etc. Display screen 240 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-oLed, or a quantum dot light-emitting diode (QLED). In some embodiments, electronic device 200 may include one or N display screens 240, where N is a positive integer greater than 1.

[0095] The electronic device 200 can implement a shooting function through an ISP, a camera 230, a video codec, a GPU, a display screen 240, and an application processor.

[0096] The ISP processes data fed back by camera 230. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and transformed into a visible image. The ISP can also perform algorithmic optimization on image noise, brightness, and skin tone. It can also optimize parameters such as exposure and color temperature of the captured scene. In some embodiments, the ISP can be located within camera 230.

[0097] The camera 230 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 200 may include 1 or N cameras 230, where N is a positive integer greater than 1.

[0098] The digital signal processor is used to process digital image signals.

[0099] Video codecs are used to compress or decompress digital video. Electronic device 200 may support one or more video codecs. This allows electronic device 200 to play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, and MPEG4.

[0100] The memory 220 can be used to store computer executable program codes, which include instructions. The memory 220 may include a program storage area and a data storage area. The program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 200 (such as audio data, a phone book, etc.), etc. In addition, the memory 220 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 210 executes various functional applications and data processing of the electronic device 200 by running instructions stored in the memory 220 and / or instructions stored in a memory provided in the processor.

[0101] After introducing the possible hardware architecture of the electronic device 200 according to the embodiment of the present application, the software system architecture of the electronic device 200 is exemplarily introduced below in conjunction with the schematic block diagram of the video recording system architecture of the electronic device 200 shown in FIG. 3 .

[0102] As shown in FIG3 , the software system architecture of the electronic device 200 includes an application layer, a framework and service layer, and a hardware abstract layer (HAL).

[0103] The application layer includes the camera app and the gallery app. The camera app can implement video recording and viewing functions. The gallery app can access videos and metadata in the file system through the media library.

[0104] The framework and service layer includes a camera service and a media library. The camera service includes a real-time recording sub-service, which may include camera devices and camera streams, etc. The media library includes modules such as the Media Store database interface, Media Provider-DB, an interception trigger module, and a segmented video startup service. The segmented video startup service includes functional modules such as scheduling strategies and task scheduling. The segmented video startup service is used to schedule algorithm tasks according to the scheduling strategy. The algorithm task may refer to each algorithm process (proc) in the segmented algorithm processing. Each algorithm process is scheduled according to the scheduling strategy to execute each algorithm process in an orderly manner. The interception trigger module is used to detect trigger conditions, and when the trigger conditions are detected, trigger the segmented startup service to perform algorithm task scheduling to schedule the execution of multiple algorithm processes to obtain 120 points of high-quality video. The trigger condition may be, for example, a user clicking on the thumbnail of a video file, or a user viewing, sharing, or editing a video file.

[0105] The HAL includes a camera HAL, an algorithm service, a segmented algorithm service, and a file system. The camera HAL includes an acquisition module for acquiring image data. The algorithm service is used to execute multiple algorithm processing to process image data online in real time to obtain 100-point basic quality video. The segmented algorithm service includes submodules such as algorithm scheduling management, segmented algorithm processing 1, and segmented processing algorithm N. The algorithm scheduling management is used to schedule and manage each processing algorithm. Segmented algorithm processing 1 and segmented processing algorithm N each include multiple algorithm processes (proc). The file system stores metadata, 100-point basic quality video, and 120-point high-quality video.

[0106] It is worth noting that in the video recording system architecture shown in FIG3 , the video processing process is exemplarily divided into two processes, stage one and stage two. Basic quality video is generated in stage one, and high-quality video is generated in stage two. In addition, stage two is also divided into multiple segmented algorithm processing processes (such as segmented algorithm processing 1 to N in FIG3 ). This is merely an example and does not constitute a limitation to the technical solution of the present application. For example, in actual applications, it is not necessary to divide stage two into multiple segmented algorithm processing processes.

[0107] Different from the system architecture in Figure 1B, the system architecture in Figure 3 adds modules for the Gallery app to perform algorithmic processing on videos with a base quality score of 100, such as the interception trigger module and segmented algorithm processing 1 to N. For users, the quality of captured videos will improve after being stored in the Gallery app for a period of time.

[0108] It is understandable that after obtaining the 120-point high-quality video, the electronic device 200 can also delete the Meta data and the 100-point basic quality video stored in the file system, and change the thumbnails and other controls originally associated with the 100-point basic quality video to be associated with the 120-point high-quality video.

[0109] In addition, the 100-point basic quality video and 120-point high-quality video are only for the convenience of expression to show that the video quality of stage two is higher than that of stage one, and do not limit the actual video quality of the video files shot in stage one and the video files obtained in stage two.

[0110] The video processing solution provided in the embodiment of the present application is further described below in combination with the hardware system architecture and software system architecture of the electronic device 200.

[0111] Referring to FIG. 4 , a schematic flow chart of a video processing method provided in an embodiment of the present application is shown. The video processing method may include:

[0112] Step S401: The electronic device 200 displays a video framing interface of a camera application.

[0113] It is understandable that the electronic device 200 can start the camera application in response to the click operation after receiving the click operation acting on the camera application, and display the video framing interface of the camera application. At this time, the electronic device 200 is in the video preview state and has not entered the video recording process.

[0114] For example, referring to FIG. 5A , which illustrates a schematic diagram of a video recording process according to an embodiment of the present application, a main interface 52 of a mobile phone 51 displays application icons such as a camera application 53, a gallery application 54, and a settings application. When a click operation is received on the camera application 53, the mobile phone 51 responds by displaying a video framing interface 55. The framing frame of the video framing interface 55 displays a preview video stream and also includes a video recording control 56 and further controls 57.

[0115] Step S402: The electronic device 200 receives a first operation on the video framing interface and enters video recording.

[0116] The first operation is an operation for triggering the electronic device 200 to enter a video recording process, which may include one or more operations.

[0117] In some embodiments, the first operation refers to an operation on a video recording control to trigger the electronic device 200 to enter a video recording process in a normal mode.

[0118] In other embodiments, the first operation may include an operation on a video recording module control and an operation on a video recording control. The operation on the video recording module control is used to trigger the electronic device 200 to determine a target recording mode, and the operation on the video recording control triggers the electronic device 200 to enter the video recording process in the target recording mode. The target recording mode refers to a specific video recording mode, which is different from the ordinary video recording mode. For example, the target recording mode may include, but is not limited to, movie mode, telephoto mode, night scene sports mode, and slow motion mode.

[0119] For example, as shown in Figure 5A, after the mobile phone 51 displays the video framing interface 55, the user can click the video recording control 56 to trigger the mobile phone 51 to enter the video recording process in normal mode; or click the more control 57 to select the video recording mode and enter the video recording process in a specific video recording mode.

[0120] When the user clicks the video recording control 56 on the video framing interface 55, the mobile phone 51 enters the video recording mode in response to the click operation on the video recording control 56 and displays the video framing interface 58 in the video recording state. At this time, the first operation is the click operation on the video recording control 56.

[0121] When the user clicks on the more controls 57 on the video framing interface 55, the mobile phone 51 displays a window 59. Window 59 includes multiple video recording modes, such as movie mode, telephoto mode, night motion mode, and slow motion mode. When the user clicks on the night motion mode control displayed in window 59, the mobile phone 51 responds to the click operation, determines that the target recording mode is the night motion mode, and displays the video framing interface 510 in the video preview state of the night motion mode; after the mobile phone 51 receives the click operation on the video recording control 56 on the video framing interface 510, it responds to the click operation, enters the video recording of the night motion mode, and displays the video framing interface 58 in the video recording state. At this time, the first operation may include a click operation on the more controls 57, a click operation on the night motion mode in window 59, and a click operation on the video recording control 56.

[0122] Step S403: During the video recording process, the electronic device 200 obtains the original image data and metadata required by the video algorithm of the second processing process, and performs the first processing process on the original image data through the image signal processor to obtain first video data.

[0123] Raw image data (also known as "raw image") refers to the image data output by an image sensor. Depending on the pixel array of the image sensor, the resulting raw image may vary accordingly. Typically, image sensors use a Bayer array, and the raw image is typically a Bayer image.

[0124] It should be pointed out that during the video recording process, in addition to collecting 3A (automatic exposure control AE, automatic focus control AF, automatic white balance AWB) information, image information (such as image width and height), and exposure parameters (aperture size, shutter speed and sensitivity aperture value) and other metadata (Mate data), the electronic device 200 also specifically collects the metadata required for the subsequent gallery application to execute the video algorithm of the second processing process.

[0125] Normally, when the electronic device 200 enters the video recording process of a specific video recording mode, the electronic device 200 can specifically collect metadata of the video algorithm corresponding to the specific video recording mode. The video algorithms required to be executed in various specific video recording modes are pre-set, and the execution order between multiple video algorithms can also be pre-set. For example, the video algorithms corresponding to the night scene motion mode include noise reduction algorithms, anti-shake algorithms and deblurring algorithms. At this time, when the electronic device 200 enters the video recording process of the night scene motion mode, the metadata collected by the electronic device 200 includes gyroscope (Gyro) information and low fps Raw images. Gyroscope information can be used for deblurring algorithms and anti-shake algorithms, and low fps Raw images can be used for noise reduction algorithms. At this time, the algorithm of the second processing process includes multiple video algorithms corresponding to the specific video recording mode.

[0126] When the electronic device 200 enters the video recording process in normal mode, the electronic device 200 can first perform scene recognition based on the raw image and collect the required metadata based on the recognized scene. For example, if the electronic device 200 can recognize that the current scene is a night scene based on the raw image data, it can specifically collect the metadata required for the night scene (such as exposure data, etc.).

[0127] It is understandable that the second processing step includes multiple video algorithms, but not all of them require metadata. That is, during the second processing step, some video algorithms require metadata, while others do not. Therefore, when collecting metadata, metadata is collected for those video algorithms that require metadata.

[0128] It is understood that after the electronic device 200 enters video recording, the raw image output by the image sensor undergoes real-time online processing by the video ISP chip to obtain first video data. That is, the ISP performs a first processing process on the raw image to obtain the first video data. The first processing process typically includes running multiple video algorithms within the video ISP chip to process the raw image.

[0129] Step S404: If the electronic device 200 receives a second operation on the video framing interface during the video recording process, it exits the video recording, obtains the first video file, stores the first video file and metadata, and updates the first video file to the gallery application. The first video file is a video file generated by encoding the first video data.

[0130] The second operation is an operation for triggering the electronic device 200 to exit video recording, and the operation may include one operation or multiple operations.

[0131] For example, referring to the schematic diagram of the video recording process shown in FIG5B , based on the scenario of FIG5A , after the mobile phone 51 enters video recording and displays the video recording viewfinder interface 58 in the video recording state, the mobile phone 51 continuously records the video. During the continuous video recording process, when the mobile phone 51 receives a click operation on the video recording control 511 on the video recording viewfinder interface 58, the mobile phone 51 exits the video recording in response to the click operation, generates a first video file, and stores the first video file and metadata in a non-volatile storage medium. In this case, the second operation is a click operation on the video recording control 511.

[0132] Among them, the video ISP chip processes each frame of Raw image in real time online. After obtaining the first video data corresponding to each frame of Raw image, the electronic device 200 can encode and package each first video data to generate a first video file when the video recording ends.

[0133] After the electronic device 200 updates the first video file to the gallery application, the gallery application displays a thumbnail of the first video file. The user can click on the thumbnail of the first video file to view, play, edit, and share the first video file. Typically, the electronic device 200 can insert a file record of the first video file into the media library. Based on the file record of the first video file, the gallery application can read the first video file from the file system, thereby updating the first video file to the gallery application.

[0134] Optionally, electronic device 200 may also display a thumbnail of the first video file on the video framing interface. For example, referring to FIG5B , after receiving a click operation on video recording control 511, mobile phone 51 exits video recording in response to the click operation and displays video framing interface 512 in a video preview state. A thumbnail 513 of the first video file is also displayed on video framing interface 512. When the user clicks thumbnail 513 of the first video file, electronic device 200 jumps from the camera application to the gallery application and displays the first video file.

[0135] Referring to Figures 2 and 3, the first video file may refer to a 100-point basic quality video, and steps S401 to S404 may refer to stage 1, i.e., the process of generating a basic quality video. In the process of generating a basic quality video, when the electronic device 200 receives a first operation, the camera application responds to the first operation and sends a recording request to the real-time recording sub-service; the real-time recording sub-service responds to the recording request and calls the camera 230 through the acquisition module of the camera HAL to perform image acquisition, obtains the Raw image output by the image sensor, and specifically collects the metadata required by the video algorithm of stage 2; the Raw image output by the acquisition module is transmitted to the algorithm processing module of the algorithm service, and the algorithm processing module sequentially executes multiple algorithm processes (i.e., video algorithms) on the Raw image in real time to obtain the first video data; when the electronic device 200 receives a second operation, it exits video recording, encodes the first video data, generates a first video file (i.e., a 100-point basic quality video), updates the 100-point basic quality video to the gallery application through the media library, and stores the 100-point basic quality video and metadata in the file system. At this point, Phase 1 is completed, a 100-point basic quality video is obtained, and the 100-point basic quality video is updated to the Gallery app.

[0136] It is worth noting that during the video recording process, the electronic device 200 processes the Raw image online in real time through the video ISP chip, and can obtain a 100-point basic quality video when the video recording ends, meeting the real-time nature of video recording.

[0137] In addition, compared with not generating a video file when ending video recording, the embodiment of the present application encodes and packages the processing results of the first processing process when ending video recording to generate a first video file. The first video file can be viewed by the user, meeting the real-time requirements of video recording and providing a higher user experience.

[0138] Step S405: The gallery application of the electronic device 200 calls the metadata to perform the second processing on the first video file according to the video algorithm of the second processing process, obtains the second video file, and stores the second video file; the video quality of the second video file is higher than the video quality of the first video file.

[0139] Among them, the second processing process includes the first type of video algorithm and / or the second type of video algorithm. The first type of video algorithm is a video algorithm that has not been executed in the first processing process, and the second type of video algorithm is a video algorithm that has been executed in the first processing process but the execution effect has not achieved the optimal target effect.

[0140] After the electronic device 200 updates the first video file to the gallery application, the gallery application can directly perform further processing on the first video file. For example, when the camera application is running in the foreground, the gallery application performs the second processing process on the first video file in the background. Of course, the gallery application can also perform further processing on the first video file at an appropriate time. The appropriate time may refer to: detecting that the camera application is closed or the camera application is running in the background, receiving the third operation acting on the first thumbnail, or the camera application is closed or the camera application is running in the background, and the preset trigger condition is met. The first thumbnail is the thumbnail of the first video file displayed on the video framing interface after receiving the second operation.

[0141] For example, referring to the video operation diagram shown in FIG5C , based on the scenario in FIG5B , the mobile phone 51 exits video recording and displays a thumbnail 513 of the first video file on the viewfinder interface 512 in the video preview state. At this time, the camera application is running in the foreground, and the mobile phone 51 needs to perform processes such as image acquisition and preview video stream processing. After the mobile phone receives a click operation on the thumbnail 513 of the first video file, it jumps to the video viewing interface 514 of the gallery application. The video viewing interface 514 includes a sharing control 515, an editing control 516, and a trigger control 517. At this time, the camera application is running in the background and the gallery application is running in the foreground, and the mobile phone 51 does not need to perform processes such as preview video stream processing. The first thumbnail is thumbnail 513, and the third operation is a click operation on thumbnail 513.

[0142] In comparison, when video recording is finished but the camera is running in the foreground, the camera application still needs to occupy computing resources and memory resources to perform image acquisition and preview video streams and other processing processes. If the gallery application executes multiple video algorithms of the second processing process to process the first video file at this time, multiple video algorithms also need to occupy computing resources and memory resources. Therefore, the computing resources and memory resources of the electronic device 200 may be highly loaded, and the processor may be in a high-load state for a long time, which may cause the electronic device 200 to experience problems such as lag, heat and high power consumption, affecting the user experience.

[0143] After the video recording is finished, the gallery application will process the first video file only after the camera application is closed, the camera application is running in the background, or a third operation is received on the first thumbnail. Since the electronic device does not need to perform image acquisition and preview video streams and other processing processes at this time, the probability of the electronic device 200 experiencing problems such as lag, heat and high power consumption is lower, thereby improving the user experience.

[0144] Furthermore, when the camera application is closed or running in the background, the electronic device 200 determines whether a preset trigger condition is detected. If the preset trigger condition is detected, the gallery application is triggered to call the metadata to process the first video file to obtain the second video file; if the preset trigger condition is not detected, it continues to determine whether the preset trigger condition is detected.

[0145] Among them, the preset trigger conditions include any one of the following: receiving a fourth operation acting on the second thumbnail or trigger control, an operation of opening the gallery application, an operation of viewing or playing the first video file, an operation of sharing the first video file, an operation of editing the first video file, the system of the electronic device is in an idle state, and the screen of the electronic device is turned off or on.

[0146] The trigger control is a control displayed in the gallery application for triggering the video algorithm processing of the first video file. For example, referring to FIG5C , the video viewing interface 514 of mobile phone 51 includes a trigger control 517. When mobile phone 51 receives a click operation (i.e., the fourth operation) on trigger control 517, it is considered that the preset trigger condition has been detected, triggering the gallery application to further process the first video file.

[0147] The second thumbnail is a thumbnail of the first video file displayed in the gallery application. For example, please refer to Figure 5C again. After the mobile phone 51 receives a click operation on the gallery application 54, it displays the album interface in response to the click operation. The album interface includes a video album control 519. After the mobile phone 51 receives a click operation on the video album control 519, it displays the video album interface 520. The video album interface 520 includes thumbnails of each video. Thumbnails 521 to 525 are exemplarily shown in Figure 5C, where thumbnail 524 is a thumbnail of the first video file, that is, the second thumbnail. After the mobile phone 51 receives a click operation on the thumbnail 524 (i.e., the fourth operation), it displays the video viewing interface 514, and deems that the preset trigger condition is detected, triggering the gallery application to process the first video file.

[0148] Please refer to Figure 5C again. The operation of opening the gallery application refers to the operation of clicking on the gallery application 54; the operation of viewing or playing the first video file refers to the operation of clicking on the first video file displayed on the video viewing interface 514; the operation of sharing the first video file refers to the operation of clicking on the sharing control 515 in the video viewing interface 514; the operation of editing the first video file refers to the operation of clicking on the editing control 516 in the video viewing interface 514.

[0149] The system of electronic device 200 being in an idle state may mean that: the processor load is lower than a preset load threshold, and the running memory is higher than a preset memory threshold. Generally, when the user is not using electronic device 200, and no application is running in the foreground of electronic device 200, and no application is resident in the background, the electronic device can be considered to be in an idle state. Specifically, this can be determined based on the processor load, running memory, and device temperature.

[0150] Of course, the preset trigger conditions are not limited to those mentioned above. For example, the preset trigger conditions may also include: uploading the first video file to the cloud, or the processor load of the electronic device 200 is lower than a preset load threshold, the running memory is higher than a preset memory threshold, and the device temperature is lower than a preset temperature threshold.

[0151] It is worth noting that when the camera application is running in the background or closed, it further determines whether the preset trigger conditions are met. Only when the preset trigger conditions are met will the gallery application further process the first video file. In this way, when the user needs to view, edit or share the captured video, the gallery application can process the first video file in time, so that the user can obtain higher quality video in time; in addition, the gallery application can also further process the first video file when the system is idle, the screen is off or the screen is off, which can further alleviate problems such as device freezes and device heating.

[0152] Typically, after capturing and obtaining a first video file, if the user needs to perform any operations on the first video file (e.g., edit, view, share, or upload to a third-party platform), electronic device 200 can promptly trigger the gallery application to further process the first video file, thereby promptly obtaining a higher-quality second video file. If the user does not need to perform any operations on the first video file, electronic device 200 can trigger the gallery application to further process the first video file when the system is idle or the total system load is low, thereby alleviating device lag and heating, and reducing device power consumption.

[0153] In the process of the gallery application calling metadata to process the first video file, it is usually necessary to execute multiple video algorithms. That is, the second processing process may include multiple video algorithms. Multiple video algorithms can be divided into multiple stages, each stage including one or more video algorithms. For example, referring to Figure 3, the segmented algorithm service includes segmented algorithm processing 1 to segmented algorithm processing N, and each segmented algorithm processing includes multiple procs (i.e., video algorithms).

[0154] The second video file can refer to the 120-point basic quality video in Figure 3, and step S405 can refer to stage two, i.e., the process of generating high-quality video. In the process of generating high-quality video, when the interception trigger module detects a trigger condition (such as when viewing, sharing or editing a video file), the segmented video startup service is triggered to start the segmented algorithm processing. The segmented video startup service performs task scheduling on the algorithm processing process of stage two, and after determining the scheduling strategy according to the system state, each segmented algorithm processing of stage two is executed using the determined scheduling strategy. After a plurality of video algorithms have been executed, the processed video data can be encoded and packaged to obtain 120 points of high-quality video (i.e., the second video file), and the 120 points of high-quality video (i.e., the second video file) is updated to the gallery application through the media library, and the 120 points of high-quality video (i.e., the second video file) is stored in the file system. The quality of the 120-point high-quality video is higher than the quality of the 100-point basic quality video.

[0155] It should be noted that when storing the second video file, the first video file in the original storage medium can be replaced with the second video file, or the second video file can be directly stored in the storage medium. The first video file and the second video file are in the storage medium at the same time, but in different storage locations.

[0156] It is worth noting that the embodiment of the present application generates a 100-point basic quality video in real time through stage one to meet the real-time requirements of video recording; further, a 120-point high-quality video is generated through stage two, so that users can obtain higher quality video files while meeting the real-time requirements. It is understandable that stage one has strong real-time requirements, so the video ISP chip usually only executes basic processing algorithms and sacrifices the algorithm execution effect to meet the demand for real-time video output, thereby obtaining a 100-point basic quality video; the real-time requirements of the second processing process (stage two) are weaker, and multiple video algorithms can be further run, and the execution effect of the video algorithm can be executed to the optimal target effect to improve the video quality of the 100-point basic quality video.

[0157] During the video recording process, the embodiment of the present application pre-collects metadata required for the subsequent gallery application and processes the original image data in real time through the ISP. When the video recording is completed, a basic quality video (i.e., a first video file) can be obtained, meeting the real-time requirements. After the first video file is updated to the gallery application, the gallery application further processes the captured basic quality video according to the pre-collected metadata to obtain a higher quality video. Therefore, while meeting the real-time requirements, users can obtain higher quality videos, thereby improving the user experience.

[0158] Based on the above embodiment, the video processing process can be divided into two stages. The first stage refers to the process of generating a first video file, and the second stage refers to the process of generating a second video file based on the first video file. In the first stage, the video ISP chip executes multiple video algorithms online in real time; in the second stage, the gallery application executes multiple video algorithms offline. The following is an introduction to another flow diagram of the video processing method shown in Figure 6. As shown in Figure 6, the video processing method may include the following steps:

[0159] Step S601: The electronic device 200 displays a video framing interface of a camera application.

[0160] Step S602: The electronic device 200 receives a first operation on the video framing interface and enters video recording.

[0161] Step S603: During the video recording process, the electronic device 200 obtains the original image data and metadata required by the video algorithm of the second processing process, and performs the first processing process on the original image data through the image signal processor to obtain first video data. The first processing process includes multiple video algorithms.

[0162] The first processing process can refer to the multiple procs (algorithm processes) executed in the algorithm processing module in Figure 3. These are processed online in real time on the video ISP chip, subject to high real-time constraints. To meet these constraints, the number of video algorithms that can be executed by the first processing process, and even the execution performance of the video algorithms, are subject to real-time constraints. Therefore, the multiple video algorithms executed by the first processing process are typically relatively basic video algorithms, such as noise reduction and sharpening algorithms.

[0163] In the embodiment of the present application, the first processing process executed by the video ISP chip can be the same as the algorithm processing shown in Figure 1B. That is, the first processing process of the embodiment of the present application can not reduce the video algorithm originally executed by the algorithm process of Figure 1B. This is because reducing the video algorithm executed by the first processing process will reduce the video quality of the first video file and affect the user experience.

[0164] In step S604, if the electronic device 200 receives a second operation on the video framing interface during video recording, it exits the video recording, obtains the first video file, stores the first video file and metadata, and updates the first video file to the gallery application. The first video file is a video file generated by encoding the first video data.

[0165] It is worth noting that compared with not generating a video file when ending video recording, the embodiment of the present application encodes and packages the processing results of the first processing process when ending video recording to generate a first video file. The first video file can be viewed by the user, meeting the real-time requirements of video recording and providing a higher user experience.

[0166] Step S605: When the electronic device 200 detects that the camera application is closed or the camera application is running in the background, and receives a third operation acting on the first thumbnail, or the camera application is closed or the camera application is running in the background and a preset trigger condition is met, the gallery application calls the metadata to perform the second processing process on the first video file according to the video algorithm of the second processing process, obtains the second video file, and stores the second video file; the second processing process includes the first type of video algorithm and / or the second type of video algorithm, the first type of video algorithm is a video algorithm that has not been executed in the first processing process, and the second type of video algorithm is a video algorithm that has been executed in the first processing process but the execution effect does not achieve the optimal target effect; the video quality of the second video file is higher than the video quality of the first video file.

[0167] It is understood that metadata includes data required by multiple video algorithms. When metadata is called to perform the second processing on the first video file, if a video algorithm requires metadata, the metadata required by the video algorithm is called to perform the video algorithm processing. If a video algorithm does not require metadata, the metadata does not need to be called when executing the video algorithm.

[0168] As mentioned above, when the first processing process executes the video algorithm, due to the high real-time constraints, the execution effect of some video algorithms cannot be optimal, and the number of video algorithms that can be executed is also very limited, which makes the quality of the first video file poor. In order to improve the quality of videos shot by users while meeting the real-time requirements, the second processing process is executed offline on the gallery application side. The real-time constraints of the second processing process are usually low. The second processing process may include video algorithms that have been executed in the first processing process but the execution effect did not achieve the target effect, that is, the second type of video algorithm, and may also include video algorithms that have not been executed in the first processing process, that is, the first type of video algorithm. Both the first type of video algorithm and the second type of video algorithm can improve video quality.

[0169] For example, the first processing step may include algorithms such as bad pixel correction, black level correction, noise reduction, lens brightness correction, automatic white balance, color interpolation, color correction, Y gamma, sharpening, and color space conversion; the second processing step may include algorithms such as anti-shake, noise reduction, and deblurring. In this case, anti-shake and deblurring algorithms are first-category video algorithms, while noise reduction is second-category video algorithms.

[0170] The above-mentioned optimal target effect refers to the desired video picture effect. Taking the noise reduction algorithm as an example, the optimal target effect can be: the video picture has less noise and rich image details. When the first processing process executes the noise reduction algorithm, due to real-time constraints, it usually only removes part of the noise in the image, but does not restore a large amount of image details lost during noise removal, resulting in a lack of image details in the obtained video picture. When the second processing process executes the noise reduction algorithm, it can not only remove the remaining noise in the image, but also restore the image details lost during denoising, thereby achieving the optimal target effect of less noise in the video picture and rich image details, thereby improving the video picture quality.

[0171] It is worth noting that the embodiment of the present application does not put the video algorithm originally belonging to the first processing process into the subsequent second processing process for execution, but does not reduce the video algorithm of the first processing process, and further executes the required video algorithm in the second processing process to improve the video quality.

[0172] Of course, in other embodiments, the number of video algorithms executed in the first processing step can be appropriately reduced, and the reduced video algorithms can be executed in the second processing step. However, in this case, the video algorithms originally belonging to the first processing step are not simply executed in the second processing step. The second processing step also executes video algorithms that were not executed in the first processing step or whose execution effect was not optimal, in order to improve video quality.

[0173] Please refer to Figure 3. The first processing process may refer to multiple procs executed in the algorithm processing module in stage one, and the second processing process may refer to the segmented algorithm processing 1 to N in the segmented algorithm service in stage two.

[0174] As can be seen from the above, in the embodiment of the present application, during the video recording process, the camera application executes the first processing process online in real time through the video ISP chip to obtain basic quality video; after the video recording is completed, the gallery application executes the second processing process offline, which may include video algorithms that have not been executed in the first processing process, and video algorithms that have been executed in the first processing process but have not achieved the optimal target effect due to real-time reasons, thereby allowing users to obtain higher quality videos while meeting real-time requirements.

[0175] Based on the above embodiment, the electronic device 200 can enter the video recording process in the normal mode or the video recording process in the specific video recording mode. The video processing process in the specific video recording mode is described below in conjunction with another flow diagram of the video processing method provided in the embodiment of the present application shown in Figure 7. As shown in Figure 7, the video processing process may include the following steps:

[0176] Step S701: The electronic device 200 displays a video framing interface of a camera application.

[0177] Step S702: The electronic device 200 receives an operation on the video framing interface and enters a target recording mode for video recording.

[0178] It is understandable that the operations acting on the video framing interface include one or more operations for triggering the electronic device 200 to enter the video recording process of the target recording mode.

[0179] Operations on the video framing interface may include operations on the video recording mode controls on the video framing interface and operations on the recording controls on the video framing interface. Operations on the video recording mode controls on the video framing interface are used to trigger the electronic device 200 to determine that the video recording mode is the target recording mode, and operations on the recording controls on the video framing interface are used to trigger the electronic device 200 to enter video recording in the target recording mode. For example, referring to FIG. 5A , a user can select from multiple video recording modes displayed in window 59 and then click on video recording control 56 to enter the video recording process in the night scene sports mode.

[0180] The target recording mode refers to a video recording mode that is different from the normal mode. For example, the target recording mode may include movie mode, telephoto mode, night scene sports mode, and slow motion mode.

[0181] Step S703: During the video recording process, the electronic device 200 obtains the original image data and metadata required by multiple target video algorithms corresponding to the target recording mode, and performs a first processing process on the original image data through the image signal processor to obtain first video data. The multiple target video algorithms are video algorithms corresponding to the target recording mode.

[0182] It should be noted that for each target recording mode, the target video algorithm to be executed by the gallery application can be pre-set. That is, the video algorithm to be executed in the second processing step of each target recording mode can be pre-set. In addition, the execution order of multiple target video algorithms corresponding to the target recording mode can also be preset.

[0183] For example, for the night scene motion mode, video tasks such as noise, image details, motion blur, and anti-shake are key to video quality. Therefore, the target video algorithms corresponding to the night scene motion mode include: deblurring, anti-shake, and noise reduction, and the algorithm execution order is: anti-shake, noise reduction, deblurring.

[0184] After entering the target recording mode, the electronic device 200 can specifically collect metadata required by the target video algorithm corresponding to the target recording mode and store it in a non-volatile storage medium. In this way, when the gallery application further processes the first video file, it can call the metadata of the target video algorithm and execute the target video algorithm based on the metadata.

[0185] It's understandable that some target video algorithms require metadata, while others don't. Therefore, when collecting metadata, the metadata required by the target video algorithms that require metadata is collected. When executing the target video algorithms, the metadata pre-collected by the target video algorithms that require metadata is used to perform video algorithm processing.

[0186] Step S704: If the electronic device 200 receives a second operation on the video framing interface during the video recording process, it exits the video recording, obtains the first video file, stores the first video file and metadata, and updates the first video file to the gallery application. The first video file is a video file generated by encoding the first video data.

[0187] Step S705, the gallery application of the electronic device 200 calls the metadata required by the target video algorithm according to the target video algorithm corresponding to the target recording mode, and executes each target video algorithm in sequence according to the algorithm execution order of each target video algorithm to perform a second processing process on the first video file, obtain a second video file, and store the second video file; the video quality of the second video file is higher than the video quality of the first video file.

[0188] It is understandable that in the second processing process, the gallery application sequentially executes multiple target video algorithms on the first video file according to the algorithm execution order. In this case, the second processing process includes multiple target video algorithms, and the target video algorithms can be the first type of video algorithms or the second type of video algorithms.

[0189] For example, for the night scene motion mode, the target video algorithms to be executed in the second processing process include: deblurring, anti-shake and noise reduction. Deblurring and anti-shake are the first type of video algorithms, and noise reduction is the second type of video algorithm.

[0190] Optionally, the electronic device 200 may trigger the gallery application to execute the second processing procedure at an appropriate time to call the metadata required by the target video algorithm, execute each target video algorithm in sequence, and obtain a second video file with higher quality.

[0191] The appropriate timing may include: detecting that the camera application is closed or running in the background, receiving the third operation on the first thumbnail, or the camera application is closed or running in the background and a preset trigger condition is met. For details, please refer to the corresponding content above and will not be repeated here.

[0192] It is worth noting that the embodiment of the present application is based on a segmented video processing architecture, which can not only enrich the video recording mode, but also enable the video obtained in the target recording mode to take into account both video quality and real-time requirements, thereby allowing users to obtain videos with better quality and richer video recording modes, thereby improving the user experience.

[0193] Specifically, the segmented video processing architecture can include a first stage and a second stage. The first stage refers to the process of real-time online processing of the raw image on the video ISP chip to generate a first video file in real time. The second stage may refer to the process of processing the first video file on the image library application to generate a second video file. For example, see stages 1 and 2 in Figure 3.

[0194] Compared to normal video recording modes, specific video recording modes typically require the execution of more video algorithms to achieve the desired effect. For example, Night Scene Motion mode may require additional anti-shake, deblurring, and noise reduction algorithms compared to normal recording modes.

[0195] When a single-segment video processing architecture (such as the system architecture shown in FIG1B ) is used to implement a specific video recording mode, the single-segment video processing architecture only has the first stage, and the video ISP chip side needs to process the Raw image online in real time. However, since the specific video recording mode requires the execution of more video algorithms, the algorithm processing time of each frame of the Raw image will be longer when the video ISP chip side processes online in real time, and thus the real-time constraints of the video recording cannot be met. In this way, when the video recording ends, the user may not be able to obtain the shot video in real time, and the real-time requirements of the video recording cannot be met. Even when the user ends the video recording, the electronic device 200 may freeze.

[0196] Furthermore, when implementing specific video recording modes using a single-stage video processing architecture, prioritizing real-time performance will cause the video ISP to sacrifice algorithm performance to meet the real-time video output requirement. This results in poor algorithm performance on the video ISP chip. While this approach can meet real-time performance constraints, it degrades video quality, resulting in poorer video quality for users.

[0197] Therefore, single-stage video processing architectures typically cannot achieve a wide range of video recording modes, nor can they balance real-time performance with video quality. Related technologies prioritize real-time video constraints to prevent user experience of lag, sacrificing the effectiveness of video algorithms. While this approach allows for a wide range of video recording modes and meets real-time requirements, the resulting video quality is poor.

[0198] In the embodiment of the present application, when a specific video recording mode is realized through a segmented video processing architecture (such as the system architecture shown in FIG3 ), the video algorithm executed in the first stage can remain unchanged, and the video algorithm corresponding to the specific video recording mode is put into the second stage for execution. In this way, there is no need to execute more video algorithms in the first stage. When the video ISP chip is processed online in real time, the algorithm processing time of each frame of the Raw image can meet the real-time constraint, so that the user can obtain the captured video when the video recording ends, and the captured video obtained in real time will not be too bad. Furthermore, the real-time constraint of the second stage is lower. The video algorithm corresponding to the specific video recording mode is executed in the second stage to realize the specific video recording mode and obtain a higher quality video.

[0199] Therefore, based on the segmented video processing architecture, it is possible to enrich specific video recording modes, and in specific video recording modes, both real-time constraints and video quality can be taken into account.

[0200] Based on the above embodiment, in the second processing process, the gallery application can execute multiple video algorithms to process the first video file to obtain a second video file with higher quality.

[0201] When executing multiple video algorithms, the gallery application can further divide the algorithms into multiple stages and execute each stage in segments. Of course, it is also possible not to divide the algorithms into multiple stages. The following will illustrate two scenarios.

[0202] When the gallery application does not divide the multiple video algorithms into multiple stages for execution, the multiple video algorithms may be executed sequentially to obtain the second video file.

[0203] For example, referring to a schematic diagram of the second processing process shown in FIG8A , after electronic device 200 obtains a first video file and updates the first video file to the gallery application, the gallery application may first unpack and decode the first video file to obtain the first video data. Algorithms 1 through N are then executed on the first video data in the order in which the multiple video algorithms are executed. Specifically, the first video algorithm is used as input to Algorithm 1 to obtain the execution result of Algorithm 1. The execution result of Algorithm 1 is used as input to Algorithm 2 to obtain the execution result of Algorithm 2. This process continues until Algorithm N is executed and the execution result of Algorithm N is obtained. The execution result of Algorithm N is then encoded and packaged to generate a second video file. This second video file can be viewed by the user.

[0204] Optionally, the gallery application may perform the second processing on the first video file only when the trigger detection module detects a trigger condition. The trigger condition may be: detecting that the camera application is closed or running in the background, receiving a third operation on the first thumbnail, or the camera application is closed or running in the background and a preset trigger condition is met.

[0205] Optionally, if the user actively triggers the gallery application to execute the second processing procedure, the electronic device 200 can execute multiple video algorithms at one time. User-initiated triggering can refer to user triggering through interactive controls. For example, referring to FIG5C , after the mobile phone 51 detects the user's click operation on the trigger control 517, it is considered that the user actively triggered the gallery application to execute the second processing procedure. If the second processing procedure is not actively triggered by the user, but the electronic device 200 automatically triggers the second processing procedure when it determines that the system is idle or the user needs to operate (view, share or edit, etc.) the video, the electronic device 200 can execute multiple video algorithms in the second processing procedure in stages. When the gallery application divides multiple video algorithms into multiple stages for execution, one stage can include one or more algorithms, and the execution order of the multiple stages is determined by the execution order of the multiple video algorithms.

[0206] For example, in target recording mode, multiple target video algorithms are associated with the target recording mode. The execution order of these target video algorithms is pre-set, and each stage includes a target video algorithm. In this case, the gallery application can sequentially execute the staged algorithm processing of each target video algorithm in accordance with the algorithm execution order based on the first video file and metadata to obtain the second video file.

[0207] Among them, the staged algorithm processing process of each target video algorithm can be as follows: detect whether the preset trigger condition is met; if the preset trigger condition is met, decode the first target video file to obtain the first target video data; if the current target video algorithm needs to use metadata, call the metadata required by the target video algorithm, execute the target video algorithm on the first target video data, and obtain the second target video data. If the current target video algorithm does not need to use metadata, the target video algorithm can be executed on the first target video data to obtain the second target video data; encode the second target video data to obtain the second target video file, and store the second target video file; detect whether the preset stop condition is met; if the preset stop condition is not met, enter the staged algorithm processing process of the next target video algorithm; if the preset stop condition is met, determine the second target video file of the current staged algorithm processing process as the second video file; the preset stop condition is to complete the execution of multiple target video algorithms.

[0208] The first target video file is the first video file or the second target video file of the previous staged algorithm processing process, and the first target video data is the video data obtained by decoding the first video data or the second target video file of the previous staged algorithm processing process.

[0209] That is, if the first target video algorithm is being processed in a staged algorithm process, the first target video file is the first video file, and the first target video data is the first video data. If the second or subsequent target video algorithms are being processed in a staged algorithm process, the first target video file is the second target video file of the previous staged algorithm process.

[0210] Optionally, when the gallery application sequentially executes each phased algorithm processing process, if the current phased algorithm processing process is completed, the second target video file of the previous phased algorithm processing process in the storage medium can be replaced with the second target video file corresponding to the current phased algorithm processing process, that is, the code stream is replaced. At this time, the gallery application can also display thumbnails of the second target video files of each phased algorithm processing process. In other words, after each phased algorithm processing process is completed, the gallery application can display the results of the phased algorithm processing process. In this way, by saving the processing results of each phased algorithm processing process, users can view the results of each phased algorithm processing process on the gallery application side, thereby improving the user experience.

[0211] Optionally, if during the staged algorithm processing of the target video algorithm, the gallery application detects that a preset interruption trigger condition is met, it stops executing the staged algorithm processing process and stores the result data that has been processed during the staged algorithm processing process. When it is detected that the preset trigger condition is met, it continues to execute the interrupted staged algorithm processing process from the interruption point based on the result data; or, it stops executing the staged algorithm processing process, stores the first target video file of the staged algorithm processing process, and re-executes the interrupted staged algorithm processing process based on the first target video file when it is detected that the preset trigger condition is met.

[0212] Optionally, the preset interrupt trigger condition includes at least one of the following: the electronic device 200 is powered off, the device temperature of the electronic device 200 is higher than a preset temperature threshold, the running memory of the electronic device 200 is lower than a preset memory threshold, and receiving an operation for instructing to terminate processing. The operation for instructing to terminate processing can be input by the user through the interactive control displayed on the electronic device 200.

[0213] For example, referring to the schematic diagram of the second processing process in stages shown in Figure 8B, the electronic device 200 responds to the user's recording operation, calls the camera to capture and record the image signal, obtains the Raw image and the metadata required for recording the video algorithm of the subsequent second processing process; the video ISP chip processes the Raw image into video data in real time, and generates a bit stream (i.e., the first video file) according to the video data when the recording ends, stores the bit stream (i.e., the first video file) and metadata, and updates the bit stream to the gallery application so that the user can view the captured video in the gallery application.

[0214] After the electronic device 200 updates the code stream to the gallery application, the gallery application may perform a second processing on the code stream to obtain a higher quality code stream.

[0215] Specifically, the gallery application performs an algorithm trigger detection step to determine whether to perform segmented algorithm processing. If an algorithm trigger condition is detected, segmented algorithm processing is performed. If no algorithm trigger condition is detected, the application continues to determine whether an algorithm trigger condition is detected. Algorithm trigger conditions can include, for example: manual triggering by the user via an interactive button; the user viewing, editing, or sharing a video; the system being idle; and uploading a video to a third-party platform.

[0216] When the algorithm triggering condition is detected, the gallery application executes the phased processing of the first algorithm. As shown in Figure 8B, the phased processing of the first algorithm includes segmented algorithm 1 data preprocessing and segmented algorithm processing. In the segmented algorithm 1 data preprocessing, the stored code stream (i.e., the first video file) is unpacked and decoded to obtain video data, and the metadata required by the current algorithm is parsed from the metadata. After obtaining the video data and the metadata required by the current algorithm based on unpacking decoding and metadata parsing, the video data and the metadata required by the current algorithm are used as algorithm inputs to execute segmented algorithm 1 for video processing to obtain the execution result of segmented algorithm 1. The execution result of segmented algorithm 1 is encoded and packaged to generate a code stream processed by segmented algorithm 1, and the code stream in the storage medium is replaced with the code stream processed by segmented algorithm 1, and the metadata is stored.

[0217] Optionally, after completing the segmented algorithm process for each algorithm, it may be determined whether a preset stop condition is satisfied. The preset stop condition is the completion of executing multiple video algorithms. In other words, it is determined whether the multiple video algorithms to be executed have been completed. If the preset stop condition is satisfied, a second video file is obtained. If the preset stop condition is not satisfied, the algorithm trigger detection is continued to execute the segmented algorithm processing of the next algorithm.

[0218] As shown in Figure 8B, after executing the process of Segmented Algorithm 1, the gallery application executes the algorithm trigger detection step again. When the algorithm trigger condition is detected, it enters the staged processing process of the second algorithm, that is, executing the data preprocessing and processing of Segmented Algorithm 2. Similar to the process of Segmented Algorithm 1, the bitstream in the storage medium is unpacked and decoded to obtain decoded video data; the metadata required by the current algorithm is parsed from the metadata in the storage medium, and based on the decoded video data and the metadata required by the current algorithm, Segmented Algorithm 2 is executed to perform video processing and obtain the execution result of Segmented Algorithm 2; the execution result of Segmented Algorithm 2 is encoded and encapsulated to generate the bitstream processed by Segmented Algorithm 2, and the bitstream in the storage medium is replaced with the bitstream processed by Segmented Algorithm 2, and the metadata is stored.

[0219] Similarly, after the processing of the segmented algorithm N is completed and it is detected that the preset stop condition is met, a second video file is obtained. At this time, the second video file is the code stream processed by the segmented algorithm N.

[0220] In comparison, when executing the second processing step, the gallery app divides the second processing step into multiple stages and executes them in phases, which can further alleviate problems such as device lag and overheating and improve the user experience. In addition, multiple video algorithms of the second processing step are executed in phases, and after each algorithm is executed, it is encoded into a video file and stored. In this way, if the executing algorithm is interrupted, there is no need to re-execute all algorithms. Instead, the interrupted algorithm can be re-executed based on the video file corresponding to the previous video algorithm.

[0221] As shown in Figure 8B , each segmented algorithm processing process includes an interrupt handling step. During the interrupt handling step, if a preset interrupt trigger condition is detected, the interrupt handling process begins. Preset interrupt trigger conditions include, but are not limited to: manual termination by the user via interactive controls, system congestion, power failure of electronic device 200, and thermal escape due to the device temperature of electronic device 200 exceeding a preset temperature threshold. The interrupt handling process is described below with reference to the schematic diagrams of the interrupt handling process shown in Figures 8C and 8D .

[0222] As shown in FIG8C , during the operation of the segmented algorithm, interrupt detection is performed in real time to detect whether the preset interrupt trigger condition is met. When it is detected that the preset interrupt trigger condition is not met, the algorithm continues to run. When it is detected that the preset interrupt trigger condition is met, the currently processed portion of the video data is saved, and the currently processed portion of the video data is encoded and packaged to obtain a partial bitstream, and the corresponding bitstream in the storage medium is replaced with the partial bitstream. When the algorithm triggers the segmented algorithm processing process the next time, the algorithm continues to run from the interruption point based on the bitstream and metadata stored in the storage medium. That is, when the algorithm is interrupted, the processed results are saved and part of the original bitstream is replaced. When the algorithm is triggered again, the algorithm continues to run from the interrupted point until it is completed.

[0223] As shown in Figure 8D, unlike Figure 8C, when the preset interrupt trigger condition is detected, the processed video data is not saved, only the original video data is saved, and the original video data is encapsulated and encoded to generate a bitstream. The original video data refers to the input of the segmented algorithm. For example, when segmented algorithm 2 is interrupted, the original video data is the result of executing segmented algorithm 1. In this case, when the algorithm is interrupted, only the original video data is saved, and when the algorithm is triggered to execute again, the interrupted algorithm will be run from the beginning.

[0224] It's worth noting that the Gallery app typically executes multiple video algorithms when the system is idle or under low load. These algorithms have a lower priority, and situations like users using electronic devices can interrupt their execution. Setting up an interruption handling mechanism can facilitate interruptions, allowing the Gallery app to better execute the multiple video algorithms in the second processing step.

[0225] As shown above, when the video recording is finished, the electronic device 200 obtains the first video file and updates the first video file to the gallery application. The gallery application can process the first video file again at an appropriate time to obtain a second video file with higher quality.

[0226] After the first video file is updated to the Gallery app, a thumbnail of the first video file will be displayed in the Gallery app. At this time, when the video recording is completed, the thumbnail of the first video file will also be displayed in the video viewfinder interface. Users can click the thumbnail of the first video file to view the captured video.

[0227] When the user clicks on the thumbnail of the first video file to play the captured video, but the electronic device 200 has not yet obtained the second video file, in some embodiments, the electronic device 200 can first play the first video file, and trigger the gallery application to process the first video file, obtain the second video file, and replace the first video file in the storage medium with the second video file. In this way, when the user clicks on the thumbnail of the first video file next time, the electronic device 200 can play the second video file. At this time, from the user's perception level, after the captured video has been placed in the gallery application for a period of time, when the user plays the video again, the video picture quality is significantly improved compared to the video picture quality of the first playback.

[0228] In other embodiments, the electronic device 200 may also immediately trigger the gallery application to execute the second processing process, and while the gallery application is executing the second processing process, the first video file is not played. In this case, from the user's perspective, after the user clicks the thumbnail of the video file, the electronic device 200 will wait for a period of time before playing the video. During this period of time, the electronic device 200 promptly triggers the gallery application to execute the second processing process on the first video file, obtains the second video file, and then plays the second video file after obtaining the second video file.

[0229] In some other embodiments, the electronic device 200 can immediately trigger the gallery application to process the first video file and display the processed video image in real time to perform "play-while-processing". The following is an introduction to another flow diagram of the video processing method provided by the embodiment of the present application, shown in Figure 9. As shown in Figure 9, the video processing method may include the following steps:

[0230] Step S901: The electronic device 200 displays a video framing interface of a camera application.

[0231] Step S902: The electronic device 200 receives an operation on the video framing interface and enters a target recording mode for video recording.

[0232] Step S903: During the video recording process, the electronic device 200 obtains the original image data and metadata required by multiple target video algorithms corresponding to the target recording mode, and performs a first processing process on the original image data through the image signal processor to obtain first video data. The target video algorithm is the video algorithm corresponding to the target recording mode.

[0233] Step S904: If the electronic device 200 receives a second operation on the video framing interface during the video recording process, it exits the video recording, obtains the first video file, stores the first video file and metadata, and updates the first video file to the gallery application. The first video file is a video file generated by encoding the first video data.

[0234] Step S905: The gallery application of the electronic device 200 displays the third thumbnail of the first video file.

[0235] For example, referring to FIG. 5C , a thumbnail 524 is displayed in the video album interface 520 , and the thumbnail 524 is the third thumbnail of the first video file.

[0236] Step S906: The electronic device 200 receives a fifth operation applied to the third thumbnail, where the fifth operation is used to trigger the gallery application to play the first video file.

[0237] For example, referring to FIG5C , a user can click on a thumbnail 524 in the video album interface 520. In response to the click, the mobile phone 51 displays the video viewing interface 514. The user then clicks on the video location in the video viewing interface 514 to trigger the mobile phone 51 to play the video corresponding to the thumbnail 524. In this case, the fifth operation may include a click operation on the thumbnail 524 and a click operation on the video viewing interface 514.

[0238] Step S907, the gallery application of the electronic device 200 responds to the fifth operation, decodes the first video file, and obtains first video data, where the first video data includes multiple image frames to be processed; for each image frame to be processed, the metadata required by the target video algorithm is called in accordance with the algorithm execution order, and each target video algorithm is executed on the image frame to be processed in turn to obtain the target image frame, and the target image frame is sent for display.

[0239] In a specific application, after the electronic device 200 obtains the first video data, it caches a preset number of to-be-processed image frames into a task queue, and sequentially performs algorithmic processing on each to-be-processed image frame cached in the task queue to obtain a target image frame for each to-be-processed image frame.

[0240] For example, if the target recording mode is Night Motion mode, the target video algorithms and the order of algorithm execution are: anti-shake, noise reduction, and deblurring. At this point, for each image frame to be processed, the anti-shake algorithm is first executed on the image frame to be processed to obtain an anti-shake image frame. A determination is then made as to whether all algorithms have been executed. If so, the target image frame is obtained and displayed. If not, the next video algorithm is executed. If not, the noise reduction algorithm is executed on the anti-shake image frame to obtain a de-noised image frame, and a determination is made as to whether all algorithms have been executed. If not, the deblurring algorithm is executed on the anti-shake image frame to obtain a deblurred image frame, and a determination is made as to whether all algorithms have been executed. At this point, if it is determined that all algorithms have been executed, the deblurred image frame is determined as the target image frame and is displayed. After the current target image frame is displayed, it is determined whether all image frames have been processed. If so, it is determined that all target image frames have been displayed, and an encoding and packaging operation is performed on the target image frame to obtain a second video file. If not, processing continues with the next image frame to be processed. This cycle continues until all target image frames to be processed have been displayed.

[0241] Step S908: After each target image frame is displayed, the electronic device 200 applies an encoding operation on each target image frame to obtain a second video file, and replaces the first video file in the storage medium with the second video file.

[0242] It's worth noting that when a user plays a first video file, it's further processed while the video is playing, and the resulting image is displayed in real time, achieving "play-while-processing" and improving the user experience. This allows users to see higher-quality video the first time they play a recorded video, providing a better user experience.

[0243] In order to better introduce the video processing solution provided by the embodiment of the present application, the video processing process schematic diagram in the night scene motion mode shown in Figures 5A and 10 is combined below, and the night scene motion mode is used as an example for introduction and explanation.

[0244] As shown in FIG5A , after the user selects the night scene motion mode option from the multiple video recording modes displayed in the window 59 of the mobile phone 51 and clicks the video recording control 56 on the video framing interface 510 , the mobile phone 51 enters the video recording process of the night scene motion mode.

[0245] As shown in Figure 10, during video recording in Night Motion mode, mobile phone 51 captures image signals through its camera. At this point, the image sensor of mobile phone 51 outputs a Raw image, which is then sent to the video ISP chip. Furthermore, based on the requirements of Night Motion mode, additional gyro information and Raw images are collected as metadata. Optionally, to save storage space, Raw images can be captured at a low frame rate (1-5 fps).

[0246] The video ISP chip processes the raw image into a full field of view (FOV) video, and when the video recording is completed, stores the full FOV raw stream and metadata in the read-only memory (ROM) of the mobile phone 51. Metadata includes raw images and gyro information. For example, the metadata in the ROM can be as shown in Figure 11. Image information such as raw images and depth maps can be stored in the Trak of the MP4 file, while non-image information such as gyro can be stored in the metadata of the MP4 file.

[0247] After the full FOV raw stream is stored in ROM, the user can view the full FOV raw stream through the gallery application. This completes the first stage of the processing process.

[0248] The gallery application of the mobile phone 51 performs algorithm trigger detection in real time to determine whether the algorithm trigger condition is detected; if the algorithm trigger condition is detected, the anti-shake algorithm, noise reduction algorithm and deblurring algorithm are executed serially in sequence.

[0249] When the stabilization algorithm is triggered, the Gallery app first preprocesses the stabilization data. Specifically, the full FOV raw stream is read from the ROM, and the gyro information is parsed from the metadata. Then, based on the full FOV raw stream and gyro information, the stabilization algorithm is executed to obtain the stabilized video. Finally, the full FOV raw stream in the ROM is replaced with the stabilized video, and the metadata continues to be stored. At this point, the user views the stabilized video in the Gallery app.

[0250] When the noise reduction algorithm is triggered, the Gallery app first performs noise reduction preprocessing. This involves decoding the stabilized video to obtain the decoded video and parsing the low-fps raw image from the metadata. The noise reduction algorithm is then applied to the decoded video and the raw image to obtain the noise-reduced video. Finally, the stabilized video in the ROM is replaced with the noise-reduced video, and the metadata continues to be stored. At this point, the user views the stabilized and noise-reduced video in the Gallery app.

[0251] It is worth noting that the Raw image includes a large amount of image details. Through the Raw image, a large amount of image details lost due to noise removal when the video ISP performs the noise reduction algorithm can be restored, thereby improving the video quality.

[0252] When the deblurring algorithm is triggered in the Gallery app, it first performs deblurring algorithm preprocessing. This involves decoding the denoised video to obtain the decoded video and parsing the gyro information from the metadata. The deblurring algorithm is then executed based on the decoded video and gyro information to obtain the deblurred video. Finally, the stabilized video in the ROM is replaced with the deblurred video. At this point, the metadata can continue to be stored or deleted. This way, the user sees the stabilized, denoised, and deblurred video in the Gallery app.

[0253] It is understandable that there will be interruption detection processing during the operation of the anti-shake algorithm, noise reduction algorithm, and deblurring algorithm. Please refer to the above for detailed introduction and will not be repeated here.

[0254] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0255] In the embodiments provided in this application, it should be understood that the disclosed devices, electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely schematic. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0256] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0257] The electronic device provided in the embodiments of the present application may include a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, a method as in any one of the above method embodiments is implemented.

[0258] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0259] An embodiment of the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0260] The present application also provides a chip system, comprising a processor coupled to a memory, and executing a computer program stored in the memory to implement the methods described in the above method embodiments. The chip system can be a single chip or a chip module composed of multiple chips.

[0261] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts that are not described or recorded in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. It should be understood that the size of the sequence numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. In addition, in the description of this application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish descriptions and should not be understood as indicating or implying relative importance. References to "one embodiment" or "some embodiments" in the description of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the phrases "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized.

[0262] Finally, it should be noted that the above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A video processing method, characterized in that: Applied to an electronic device, the electronic device is installed with a camera application and a gallery application; the method comprises: Displaying the video framing interface of the camera application; receiving a first operation on the video framing interface and entering video recording; During the video recording process, original image data and metadata required by a video algorithm of a second processing process are obtained, and a first processing process is performed on the original image data by an image signal processor to obtain first video data; If a second operation acting on the video framing interface is received during the video recording process, the video recording is exited, a first video file is obtained, the first video file and the metadata are stored, and the first video file is updated to the gallery application, where the first video file is a video file generated by encoding the first video data; The gallery application calls the metadata to perform a second processing process on the first video file according to the video algorithm of the second processing process, obtains a second video file, and stores the second video file; the video quality of the second video file is higher than the video quality of the first video file; Among them, the second processing process includes a first type of video algorithm and / or a second type of video algorithm, the first type of video algorithm is a video algorithm that has not been executed in the first processing process, and the second type of video algorithm is a video algorithm that has been executed in the first processing process but the execution effect does not achieve the optimal target effect.

2. The method according to claim 1, characterized in that The gallery application calls the metadata to perform a second processing process on the first video file according to the video algorithm of the second processing process to obtain a second video file, including: If the camera application is closed, running in the background, or receives a third operation on the first thumbnail, the gallery application calls the metadata to perform a second processing process on the first video file according to the video algorithm of the second processing process to obtain a second video file, and the first thumbnail is the thumbnail of the first video file displayed on the video recording interface after receiving the second operation.

3. The method according to claim 2, characterized in that If the camera application is closed or the camera application is running in the background, the gallery application calls the metadata to perform a second processing process on the first video file according to the video algorithm of the second processing process to obtain a second video file, including: If the camera application is closed or is running in the background and a preset trigger condition is detected, the gallery application calls the metadata to perform a second processing process on the first video file according to the video algorithm of the second processing process to obtain the second video file; The preset trigger condition includes any one of the following: receiving a fourth operation acting on the second thumbnail or the trigger control, an operation of opening the gallery application, an operation of viewing or playing the first video file, an operation of sharing the first video file, an operation of editing the first video file, the system of the electronic device is in an idle state, and the screen of the electronic device is turned off or off; The second thumbnail is a thumbnail of the first video file displayed in the gallery application; the trigger control is a control displayed in the gallery application for triggering video algorithm processing of the first video file.

4. The method according to any one of claims 1 to 3, characterized in that: The receiving a first operation on the video framing interface and entering video recording includes: receiving an operation of a video recording mode control on the video framing interface, and determining that the video recording mode is a target recording mode; An operation of a recording control acting on the video framing interface is received to enter video recording of the target recording mode; wherein the first operation includes the operation acting on the video recording mode control and the operation acting on the recording control.

5. The method according to claim 4, characterized in that The gallery application calls the metadata to perform the second processing on the first video file according to the video algorithm of the second processing to obtain the second video file, including: The gallery application sequentially executes the phased algorithm processing of each target video algorithm in accordance with the algorithm execution order of the multiple target video algorithms corresponding to the target recording mode based on the first video file and the metadata to obtain the second video file; the video algorithm of the second processing process includes the multiple target video algorithms, and the target video algorithm is the first type of video algorithm or the second type of video algorithm; The phased algorithm processing process of each target video algorithm includes: Detecting whether the preset trigger condition is met; If the preset trigger condition is met, a decoding operation is performed on the first target video file to obtain first target video data; the target video algorithm is executed on the first target video data to obtain second target video data, or the metadata required by the target video algorithm is called to execute the target video algorithm on the first target video data to obtain second target video data; an encoding operation is performed on the second target video data to obtain a second target video file, and the second target video file is stored; Detect whether the preset stop condition is met; If the preset stop condition is not met, then entering the next staged algorithm processing process of the target video algorithm; If the preset stop condition is met, determining the second target video file of the current staged algorithm processing process as the second video file; The preset stop condition is that the plurality of target video algorithms are executed; The first target video file is the first video file or the second target video file of the previous staged algorithm processing process, and the first target video data is the video data obtained by decoding the first video data or the second target video file of the previous staged algorithm processing process.

6. The method according to claim 5, characterized in that The method further comprises: If during the staged algorithm processing of executing the target video algorithm, it is detected that a preset interrupt trigger condition is met; Stop executing the staged algorithm processing process, and store the result data that has been processed in the staged algorithm processing process, and when it is detected that the preset trigger condition is met, continue to execute the interrupted staged algorithm processing process from the interruption point based on the result data; or stop executing the staged algorithm processing process, store the first target video file of the staged algorithm processing process, and when it is detected that the preset trigger condition is met, re-execute the interrupted staged algorithm processing process based on the first target video file.

7. The method according to claim 6, characterized in that The preset interrupt trigger condition includes at least one of the following: the electronic device is powered off, the device temperature of the electronic device is higher than a preset temperature threshold, the running memory of the electronic device is lower than a preset memory threshold, and receiving an operation for indicating termination of processing.

8. The method according to claim 5, characterized in that The method further comprises: When executing each of the phased algorithm processing processes in sequence, if the current phased algorithm processing process is completed, the second target video file of the previous phased algorithm processing process in the storage medium is replaced with the second target video file corresponding to the current phased algorithm processing process; The gallery application displays thumbnails of the second target video files during each phase of the algorithm processing.

9. The method according to claim 4, characterized in that The method further comprises: The gallery application displays a third thumbnail of the first video file; The gallery application calls the metadata to perform the second processing on the first video file according to the video algorithm of the second processing to obtain the second video file, including: receiving a fifth operation applied to the third thumbnail, wherein the fifth operation is used to trigger the gallery application to play the first video file; In response to the fifth operation, the gallery application decodes the first video file to obtain first video data, where the first video data includes a plurality of image frames to be processed; for each of the image frames to be processed, in accordance with the algorithm execution order, the metadata required by the target video algorithm is called, each of the target video algorithms is sequentially executed on the image frames to be processed, a target image frame is obtained, and the target image frame is sent for display; After each of the target image frames is displayed, the gallery application performs encoding operations on each of the target image frames to obtain the second video file, and replaces the first video file in the storage medium with the second video file.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 9 is implemented.

11. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images