Video rendering method and electronic equipment

Abstract

The invention provides a video rendering method and electronic equipment, relates to the technical field of terminals, and can store a first unrendered video and a first rendering parameter obtained in a first recording mode, process the stored first unrendered video and the first rendering parameter, and display the processed first unrendered video and the processed first rendering parameter in a second recording mode. According to the method, the second unrendered video with higher video quality and the second rendering parameter with better rendering effect are obtained, the rendered video with higher video quality and better display effect than the preview video can be generated subsequently, the requirement of the user for the high-quality video is met, and the use experience of the user is improved. Wherein the first unrendered video and the first rendering parameter are used for generating a preview video of the first unrendered video.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to a video rendering method and electronic device. Background Technology

[0002] The video recording function of the terminal device provides users with a variety of different recording modes, such as movie mode and sports mode. Different recording modes have corresponding video effects, such as video stabilization and video blurring.

[0003] Currently, video effects are typically rendered using preview rendering algorithms at the end of recording. For example, videos recorded in movie mode are rendered using preview rendering algorithms for video stabilization and video blurring, resulting in a preview video that is then displayed and saved in the gallery. However, the quality of the video effects is not high, which affects the user experience. Summary of the Invention

[0004] This application provides a video rendering method and electronic device that can optimize the unrendered video and rendering parameters used to generate a preview video, so as to generate a rendered video whose display effect is more in line with the recording mode than the preview video based on the processed unrendered video and rendering parameters, thereby improving the video quality and display effect of the rendered video and meeting the user's experience.

[0005] To achieve the above objectives, the embodiments of this application adopt the following technical solutions:

[0006] In a first aspect, a video rendering method is provided for use in an electronic device. In this video rendering method, firstly, when the electronic device is in a first recording mode, the electronic device acquires a first unrendered video and first rendering parameters. Then, the electronic device saves the first unrendered video and the first rendering parameters. Finally, when the electronic device meets a first preset condition, the electronic device processes the first unrendered video and the first rendering parameters to obtain a second unrendered video and a second rendering parameter.

[0007] The first unrendered video and the first rendering parameters are used to generate a preview video of the first unrendered video.

[0008] The first rendering parameter is determined according to the first recording mode.

[0009] For example, the first recording mode can be any one of the following: movie mode, sports mode, night mode, portrait mode, slow motion mode, time-lapse mode, panorama mode, high dynamic range imaging (HDR) mode, standard mode, or professional mode.

[0010] For example, the first preset condition includes any one of the following: the user triggers a control to render the first unrendered video, the user triggers a control to edit the first unrendered video, or the electronic device is in an idle state. Therefore, the first preset condition can be used to optimize the first unrendered video on demand, meeting the user's video processing needs and improving the user experience.

[0011] The second unrendered video has a higher video quality than the first unrendered video.

[0012] For example, video quality is determined based on at least one of resolution, frame rate, bit rate, compression ratio, distortion, and noise. Therefore, by processing the first unrendered video to obtain the second unrendered video, the video quality of the second unrendered video can be improved.

[0013] The second unrendered video and the second rendering parameters are used to obtain the rendered video.

[0014] In the above video rendering method, the first unrendered video and the first rendering parameters used to generate the preview video are stored. Under the first preset conditions of the electronic device, the first unrendered video and the first rendering parameters are processed to obtain a second unrendered video with higher video quality and a second rendering parameter with better rendering effect. This improves the video quality and display effect of the subsequently rendered video, meets the user's demand for high-quality video, and improves the user experience.

[0015] In one possible implementation of the first aspect, when the electronic device meets the second preset conditions, the electronic device obtains a rendered video based on the second unrendered video and the second rendering parameters.

[0016] Among them, the degree to which the display effect of the rendered video matches the first recording mode is greater than the degree to which the display effect of the preview video matches the first recording mode.

[0017] For example, the display effects include at least one of video stabilization, video blurring, audio enhancement, filter modes, and watermark modes. It is evident that by using the processed second unrendered video and the processed second rendering parameters, a rendered video with superior display effects compared to the preview video can be generated. The generated rendered video better meets the user's display requirements for the first recording mode, satisfies the user's demand for high-quality video rendering, and improves the user experience.

[0018] In one possible implementation of the first aspect, the second preset condition includes any one of the following: a user viewing a second unrendered video, a user sharing a second unrendered video, a user uploading a second unrendered video, a user triggering a control to render the second unrendered video, a user triggering a control to edit the second unrendered video, or the electronic device being in an idle state.

[0019] For example, when the electronic device is idle, the first preset condition can be triggered first. It can be seen that the second unrendered video can be rendered on demand using the second rendering parameters through the second preset condition, so as to meet the user's needs for video processing and improve the user experience.

[0020] In one possible implementation of the first aspect, when the electronic device obtains the second rendering parameters, it first acquires the editing parameters obtained by the user on the first unrendered video. Then, the electronic device processes the first rendering parameters according to the editing parameters to obtain the second rendering parameters. It is evident that the electronic device can optimize the first rendering parameters through user editing, meeting the processing needs of different users for rendered videos and improving the user experience.

[0021] In one possible implementation of the first aspect, when the electronic device acquires the first unrendered video, it first acquires at least one video frame image recorded in a first recording mode; then, it acquires an unrendered image of each video frame image after image processing; finally, it acquires the first unrendered video based on the unrendered image. It is evident that the first unrendered video is a video processed frame-by-frame from the video recorded by the electronic device, which can further improve the video quality compared to the video recorded by the electronic device.

[0022] In one possible implementation of the first aspect, when the electronic device obtains the first rendering parameters, it first obtains the recording parameters corresponding to the first recording mode. Then, based on the recording parameters corresponding to the first recording mode, the electronic device obtains the first rendering parameters corresponding to the first recording mode. The recording parameters include image data of each unrendered image in the first unrendered video and camera parameters when the electronic device records in the first recording mode. Therefore, by using the image data and camera parameters, first rendering parameters that better match the first recording mode can be obtained, improving the display effect of the generated preview video.

[0023] In one possible implementation of the first aspect, when the electronic device obtains the second unrendered video, it first acquires at least one unrendered image of the first unrendered video, and then processes the at least one unrendered image to obtain the second unrendered video. It is evident that the second unrendered video is a video after the electronic device has optimized the first unrendered video frame by frame, which can further improve the video quality compared to the first unrendered video.

[0024] In one possible implementation of the first aspect, when the electronic device obtains the rendered video, it acquires at least one video frame image of the second unrendered video. Then, the electronic device renders the at least one video frame image according to the second rendering parameters to obtain the rendered video. It is evident that when rendering the display effect of the second unrendered video using the second rendering parameters, rendering is also performed frame by frame. When the user views the video, the video frame images rendered frame by frame can be displayed, improving the real-time performance of the rendered video display and satisfying the user's experience.

[0025] Secondly, a video rendering method is provided for use in an electronic device. In this video rendering method, firstly, when the electronic device is in a first recording mode, the electronic device acquires a first unrendered video and first rendering parameters. Then, the electronic device saves the first unrendered video and the first rendering parameters. Finally, when the electronic device meets a second preset condition, the electronic device obtains a rendered video based on the first unrendered video and the first rendering parameters.

[0026] The first unrendered video and the first rendering parameters are used to generate a preview video of the first unrendered video; the first rendering parameters are determined according to the first recording mode.

[0027] Among them, the degree to which the display effect of the rendered video matches the first recording mode is greater than the degree to which the display effect of the preview video matches the first recording mode.

[0028] In the above video rendering method, under the second preset condition, the electronic device generates a rendered video based on the saved first unrendered video and the first rendering parameters. This is not affected by the real-time performance and power consumption of the preview video display, and can generate a rendered video with better display effect, thus meeting the user's needs for the display effect of the rendered video and improving the user experience.

[0029] In one possible implementation of the second aspect, when the electronic device obtains a rendered video based on the first unrendered video and the first rendering parameters, provided that the second preset condition is met, firstly, the electronic device processes the first unrendered video to obtain a second unrendered video, then the electronic device updates the first unrendered video based on the second unrendered video, and finally, when the electronic device meets the second preset condition, the electronic device obtains the rendered video based on the updated first unrendered video and the first rendering parameters.

[0030] The second unrendered video has a higher video quality than the first unrendered video. This demonstrates that electronic devices can optimize the saved first unrendered video to obtain a second unrendered video with higher quality, thereby improving the overall video quality of the rendered video.

[0031] In one possible implementation of the second aspect, when the electronic device obtains a rendered video based on the updated first unrendered video and the first rendering parameters, provided that the second preset conditions are met, the electronic device first processes the first rendering parameters to obtain second rendering parameters. Then, the electronic device updates the first rendering parameters based on the second rendering parameters. Finally, when the electronic device meets the second preset conditions, it obtains the rendered video based on the updated first unrendered video and the updated first rendering parameters. It is evident that the electronic device can optimize the saved first rendering parameters to obtain second rendering parameters with better rendering effects. Based on the second unrendered video with higher video quality and the second rendering parameters with better rendering effects, a rendered video with higher video quality and better display effects can be generated.

[0032] Thirdly, an electronic device is provided, the electronic device including a memory and one or more processors; the memory is coupled to the processors; wherein the memory stores computer program code, the computer program code including computer instructions, and when the computer instructions are executed by the processor, the electronic device performs the video rendering method as described in the first aspect and any implementation thereof.

[0033] Fourthly, a computer-readable storage medium is provided, including computer instructions that, when executed on an electronic device, cause the electronic device to perform a video rendering method as described in the first aspect and any implementation thereof.

[0034] Fifthly, a computer program product is provided that, when run on an electronic device, causes the electronic device to execute the video rendering method as described in the first aspect and any of its implementations.

[0035] The beneficial effects that the electronic equipment provided in the third aspect, the computer-readable storage medium provided in the fourth aspect, and the computer program product provided in the fifth aspect can achieve can be referred to the beneficial effects that the first aspect and any of its implementations, and the second aspect and any of its implementations can achieve, and will not be repeated here. Attached Figure Description

[0036] Figure 1 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application is shown;

[0037] Figure 2 A schematic diagram of the software structure of an electronic device provided in an embodiment of this application is shown;

[0038] Figure 3 A flowchart of a video rendering method provided in an embodiment of this application is shown;

[0039] Figure 4 This illustration shows a schematic diagram of a mobile phone recording video in a first recording mode, according to an embodiment of this application.

[0040] Figure 5 This document illustrates a flowchart of a method for generating a preview video, as provided in an embodiment of this application.

[0041] Figure 6 This document illustrates a flowchart of a video optimization method in movie mode provided by an embodiment of this application.

[0042] Figure 7 A flowchart illustrating an optimized video provided in an embodiment of this application is shown;

[0043] Figure 8 This document illustrates a flowchart of a method for generating a rendered video according to an embodiment of this application.

[0044] Figure 9 A flowchart illustrating a frame-by-frame rendering method provided in an embodiment of this application is shown;

[0045] Figure 10 A schematic diagram of a first unrendered video provided in an embodiment of this application is shown;

[0046] Figure 11 A schematic diagram of a first rendering parameter provided in an embodiment of this application is shown;

[0047] Figure 12 A schematic diagram of a second unrendered video provided in an embodiment of this application is shown;

[0048] Figure 13 A schematic diagram of a second rendering parameter provided in an embodiment of this application is shown;

[0049] Figure 14A schematic diagram of a video rendering method provided in an embodiment of this application is shown;

[0050] Figure 15 A schematic diagram of the hardware structure of another electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0051] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. In the description of this application, unless otherwise stated, " / " indicates that the objects before and after are in an "or" relationship. For example, A / B can represent A or B. "And / or" in this application is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone, where A and B can be singular or plural. Furthermore, in the description of this application, unless otherwise stated, "multiple" refers to two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple. Furthermore, to facilitate a clear description of the technical solutions in the embodiments of this application, the terms "first" and "second" are used in the embodiments of this application to distinguish identical or similar items with substantially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that "first" and "second" are not necessarily different. Meanwhile, in the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is being used as an example, illustration, or description. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner for ease of understanding.

[0052] Furthermore, the business scenarios described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the emergence of new business scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0053] With the continuous development of video functions in terminal devices, these devices offer users more options for video shooting. For example, they provide various recording modes, such as movie mode and sports mode. Different recording modes have corresponding video effects, such as video stabilization, video blurring, filters, watermarks, and audio enhancement. After recording is completed, the recorded video needs to be rendered with video effects before it can be displayed to the user.

[0054] Currently, the recorded video is typically processed frame by frame using image signal processing (ISP). Then, a preview rendering algorithm is used to render the ISP-processed video to obtain a preview video for display and storage in the gallery. However, the video quality saved in this way is not high due to the constraints of real-time preview. If the video quality is to be improved, a large amount of neural processing unit (NPU) resources are required when generating the preview video, resulting in high power consumption of the terminal device. This not only affects the real-time performance of the preview but also the performance of the terminal device, making it impossible to simultaneously meet the requirements of video quality and real-time video preview.

[0055] In view of this, this application provides a video rendering method. Taking its application in an electronic device as an example, firstly, the electronic device obtains a video that has not yet been rendered with video effects and rendering parameters from the local device. Then, the electronic device renders the video that has not yet been rendered with video effects using a rendering algorithm based on the rendering parameters to obtain a rendered video. The rendering algorithm used by the electronic device locally is superior to the preview rendering algorithm used by the electronic device when generating a preview video in recording mode. In this way, the electronic device can improve the quality of the rendered video by using a better rendering algorithm locally without affecting the real-time display of the preview video, thereby improving the user experience.

[0056] In some embodiments, firstly, while in a first recording mode, the electronic device acquires a first unrendered video and first rendering parameters, wherein the first unrendered video and first rendering parameters are used to generate a preview video of the first unrendered video; the first rendering parameters are determined according to the first recording mode; then, the electronic device saves the first unrendered video and first rendering parameters; finally, if the electronic device meets a first preset condition, the electronic device processes the first unrendered video and first rendering parameters to obtain a second unrendered video and second rendering parameters, wherein the video quality of the second unrendered video is higher than that of the first unrendered video, and the second unrendered video and second rendering parameters are used to obtain a rendered video.

[0057] For example, the first recording mode could be movie mode, motion mode, delay mode, or slow motion mode, etc.

[0058] For example, the first preset condition includes any one of the following: the user triggers a control to render the first unrendered video, the user triggers a control to edit the first unrendered video, or the electronic device is in an idle state.

[0059] For example, video quality is determined based on at least one of resolution, frame rate, bitrate, compression ratio, distortion, and noise. Generally, high resolution, high frame rate, appropriate bitrate, low compression ratio, low distortion, and low noise indicate higher video quality. For instance, when evaluating video quality, metrics such as resolution, frame rate, bitrate, compression ratio, distortion, and noise can be substituted into the video quality evaluation formula; the higher the evaluation value, the higher the video quality. Professional video quality evaluation tools and methods can also be used, such as objective evaluation metrics like peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) to quantify video quality. Furthermore, video quality can also be evaluated subjectively, such as by assessing the quality of a video through the observer's visual perception.

[0060] In the above method, the electronic device saves the first unrendered video and the first rendering parameters acquired in the first recording mode, and optimizes the saved first unrendered video and the first rendering parameters to improve the video quality of the subsequently rendered video, while not affecting the real-time display of the preview video.

[0061] In some embodiments, firstly, while in a first recording mode, the electronic device acquires a first unrendered video and first rendering parameters, wherein the first unrendered video and first rendering parameters are used to generate a preview video of the first unrendered video; the first rendering parameters are determined according to the first recording mode; then, the electronic device saves the first unrendered video and the first rendering parameters; finally, when the electronic device meets a second preset condition, the electronic device obtains a rendered video based on the first unrendered video and the first rendering parameters; wherein the degree of matching between the display effect of the rendered video and the first recording mode is greater than the degree of matching between the display effect of the preview video and the first recording mode.

[0062] For example, the second preset condition includes any one of the following: the user views the first unrendered video, the user shares the first unrendered video, the user uploads the first unrendered video, the user triggers a control to render the first unrendered video, the user triggers a control to edit the first unrendered video, or the electronic device is in an idle state.

[0063] For example, the display effects include at least one of video stabilization, video blurring, audio enhancement, filter mode, and watermark mode.

[0064] In the above method, under the second preset conditions, the electronic device obtains a rendered video based on the stored first unrendered video and the first rendering parameters. It is not constrained by the real-time display of the preview video and can generate a rendered video with a better display effect than the preview video.

[0065] In some embodiments, firstly, while in a first recording mode, the electronic device acquires a first unrendered video and first rendering parameters, wherein the first unrendered video and first rendering parameters are used to generate a preview video of the first unrendered video; the first rendering parameters are determined according to the first recording mode; then, the electronic device saves the first unrendered video and first rendering parameters; secondly, if the electronic device meets a first preset condition, the electronic device processes the first unrendered video and first rendering parameters to obtain a second unrendered video and second unrendered parameters; wherein the video quality of the second unrendered video is higher than that of the first unrendered video; finally, if the electronic device meets a second preset condition, the electronic device obtains a rendered video based on the second unrendered video and second rendering parameters; wherein the display effect of the rendered video matches the first recording mode to a greater extent than the display effect of the preview video matches the first recording mode.

[0066] For example, software or plugins with automatic evaluation functions can be used to evaluate the degree of matching between the display effects of the rendered video and the preview video and the first recording mode, respectively. Alternatively, subjective evaluation can be used to evaluate the degree of matching between the display effects of the rendered video and the preview video and the first recording mode, respectively. When using software with automatic evaluation functions for evaluation, the display effect of video stabilization can be reflected by parameters such as smoothness and jitter suppression rate provided by the software. Alternatively, the display effect of video stabilization can be evaluated by detecting motion changes between video frames of the rendered video and between video frames of the preview video. Additionally, the display effect of video blurring can be reflected by parameters such as sharpness and contrast difference between foreground and background provided by the software. For example, the first recording mode is movie mode. Movie mode requires image stabilization and shallow depth of field (blurring) to enhance the cinematic feel. Software evaluation shows that the smoothness and jitter suppression rate of the rendered video are higher than those of the preview video. The sharpness and contrast difference between the foreground and background of the rendered video are more in line with the requirements of the first recording mode than those of the preview video. Visually, subjective evaluation shows that the cinematic feel of the rendered video is also stronger than that of the preview video. This indicates that the display effect of the rendered video matches the first recording mode better than the display effect of the preview video.

[0067] In the above method, under the first preset condition, the electronic device optimizes the stored first unrendered video and the first rendering parameters to obtain a second unrendered video with higher video quality and a second rendering parameter with better rendering effect. Based on the second unrendered video and the second rendering parameter, a rendered video with higher video quality and better display effect is obtained, which can simultaneously take into account the real-time display of the preview video and the high-quality rendered video, thereby improving the user experience.

[0068] The video rendering method provided in this application embodiment can be applied to electronic devices.

[0069] In some embodiments, such as Figure 1 As shown, the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc.

[0070] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0071] Processor 110 may include one or more processing units ( Figure 1 (Not shown in the image), for example, processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0072] The controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to the instruction opcode and timing signals to complete the control of fetching and executing instructions.

[0073] In some possible implementations, the processor 110 may specifically control the electronic device 100 to acquire and save the first unrendered video and the first rendering parameters in the first recording mode. The processor 110 may also specifically control the electronic device 100 to optimize the first unrendered video and the first rendering parameters under the condition of satisfying the first preset condition, so as to obtain the second unrendered video and the second rendering parameters.

[0074] In some possible implementations, after the processor 110 controls the electronic device 100 to obtain the second unrendered video and the second rendering parameters, the processor 110 can also specifically control the electronic device 100 to obtain the rendered video based on the second unrendered video and the second rendering parameters when the second preset conditions are met; the degree of matching between the display effect of the rendered video and the first recording mode is greater than the degree of matching between the display effect of the preview video and the first recording mode.

[0075] In some possible implementations, after the processor 110 controls the electronic device 100 to save the first unrendered video and the first rendering parameters, the processor 110 can also specifically control the electronic device 100 to obtain the rendered video based on the first unrendered video and the first rendering parameters under the second preset conditions; the degree of matching between the display effect of the rendered video and the first recording mode is also greater than the degree of matching between the display effect of the preview video and the first recording mode.

[0076] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0077] In some embodiments, processor 110 may include one or more interfaces ( Figure 1(Not shown in the image). For example, USB interface 130 is an interface conforming to the USB standard specification, specifically a Mini USB interface, Micro USB interface, USB Type-C interface, etc. USB interface 130 can be used to connect a charger to charge electronic device 100, and can also be used for data transfer between electronic device 100 and peripheral devices. It can also be used to connect headphones for audio playback. This interface can also be used to connect other electronic devices, such as AR devices.

[0078] The charging management module 140 is used to receive charging input from the charger. The charger can be a wireless charger or a wired charger.

[0079] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to power the processor 110, internal memory 121, external memory, display 194, camera 193, and wireless communication module 160, etc.

[0080] The wireless communication function of electronic device 100 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0081] Antennas 1 and 2 are used to transmit and receive electromagnetic wave signals. The mobile communication module 150 can provide wireless communication solutions, including 2G / 3G / 4G / 5G, for use on the electronic device 100. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1.

[0082] The wireless communication module 160 can provide solutions for wireless communication applications on the electronic device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 receives electromagnetic waves via antenna 2, modulates and filters the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, modulate and amplify them, and then convert them into electromagnetic waves for radiation via antenna 2.

[0083] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, so that electronic device 100 can communicate with networks and other devices through wireless communication technology.

[0084] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU performs mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information. Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel.

[0085] In some possible implementations, the electronic device 100 can display a preview video or a rendered video via a display screen 194.

[0086] It is understood that the display screen 194 mentioned herein can serve as the screen in the following embodiments.

[0087] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0088] The ISP is used to process the data fed back by camera 193. Camera 193 is used to capture still images or videos.

[0089] In some possible implementations, the video of the electronic device 100 in the first recording mode is recorded by the camera 193, and the video recorded by the electronic device 100 in the first recording mode is processed by the ISP to obtain the first unrendered video.

[0090] A digital signal processor (DSP) is used to process digital signals, including digital image signals and other digital signals. A video codec is used to compress or decompress digital video. An NPU (Neural Processing Unit) is a neural network (NN) computing processor that, by borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, rapidly processes input information and can continuously learn. NPUs enable intelligent cognitive applications in electronic devices, such as image recognition, speech recognition, and text understanding.

[0091] In some possible implementations, the NPU performs optimization processing on the first unrendered video, such as video analysis, video enhancement, or video encoding / decoding, to obtain the second unrendered video. For example, video analysis can analyze key information in the video stream of the first unrendered video in real time, such as people, animals, buildings, or objects, so as to optimize for different key information. Video enhancement can perform real-time enhancement processing on the first unrendered video, such as color correction, noise suppression, and super-resolution. Video encoding / decoding can accelerate the encoding / decoding process of the first unrendered video, improving the storage efficiency and subsequent processing (optimization or rendering) efficiency of the first unrendered video. Video encoding / decoding can also accelerate the encoding / decoding process of the second unrendered video, improving the storage efficiency and subsequent rendering efficiency of the second unrendered video.

[0092] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to perform data storage functions.

[0093] In some possible implementations, the electronic device 100 can store a first unrendered video and a first rendering parameter via an external memory card, the electronic device 100 can also store a second unrendered video and a second rendering parameter via an external memory card, and the electronic device 100 can also store a rendered video via an external memory card.

[0094] Internal memory 121 can be used to store computer executable program code, which includes instructions. Processor 110 executes various functional applications and data processing of electronic device 100 by running the instructions stored in internal memory 121. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of electronic device 100 (such as audio data, phonebook, etc.). Furthermore, internal memory 121 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0095] In some possible implementations, the electronic device 100 may also store the first unrendered video and the first rendering parameters through the internal memory 121, the electronic device 100 may also store the second unrendered video and the second rendering parameters through the internal memory card, and the electronic device 100 may also store the rendered video through the internal memory card.

[0096] Electronic device 100 can implement audio functions through audio module 170 and application processor, such as music playback and recording. Audio module 170 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. Audio module 170 can also be used for encoding and decoding audio signals.

[0097] Buttons 190 include a power button, volume buttons, etc. Buttons 190 can be mechanical buttons or touch-sensitive buttons. Motor 191 can generate vibration alerts. Motor 191 can be used for incoming call vibration alerts or for touch vibration feedback. Indicator 192 can be an indicator light, used to indicate charging status, battery level changes, or to indicate messages, missed calls, notifications, etc. SIM card interface 195 is used to connect a SIM card.

[0098] based on Figure 1 The electronic device shown implements the video rendering method in this application embodiment. It can store a first unrendered video and a first rendering parameter used by the electronic device to generate a preview video in a first recording mode. The stored first unrendered video and the first rendering parameter are optimized to obtain a second unrendered video and a second rendering parameter, which facilitates the generation of a rendered video. At the same time, it takes into account both the real-time requirements of the preview video and the video quality requirements of the rendered video, thereby improving the user experience.

[0099] The software system of electronic device 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment uses a layered mobile operating system as an example to exemplify the software structure of electronic device 100.

[0100] like Figure 2 As shown, the layered architecture divides the software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the mobile operating system is divided into four layers, from top to bottom: the application layer, the application framework layer, the system service layer, and the kernel layer.

[0101] The application layer can include a series of application packages.

[0102] like Figure 2 As shown, the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, and SMS.

[0103] In some embodiments, the application layer of the electronic device 100 may include an application that optimizes the first unrendered video and the first rendering parameters to generate a second unrendered video and a second rendering parameter, and an application that generates a rendered video based on the second unrendered video and the second rendering parameter. Taking the electronic device 100 as a mobile phone as an example, the application may be a pre-installed system-level application or an application downloaded by the user during use as needed. This application embodiment does not specifically limit this.

[0104] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0105] like Figure 2 As shown, the application framework layer may include a window manager, content provider, view system, phone manager, resource manager, notification manager, etc.

[0106] The view system includes visual controls, such as controls for displaying text and controls for displaying images. The view system can be used to build applications. The display interface can consist of one or more views.

[0107] The system service layer can include multiple functional modules. For example: surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), 2D graphics engines (e.g., SGL), etc.

[0108] In some embodiments, the system service layer of the electronic device 100 may further include a video rendering module. The video rendering module is used to acquire a first unrendered video and first rendering parameters when the electronic device is in a first recording mode. The first unrendered video and first rendering parameters are used to generate a preview video of the first unrendered video. The first rendering parameters are determined according to the first recording mode. The first unrendered video and first rendering parameters are stored in the memory of the electronic device. When the electronic device meets a first preset condition, the first unrendered video and first rendering parameters are optimized to obtain a second unrendered video and second rendering parameters. The video quality of the second unrendered video is higher than that of the first unrendered video. Subsequently, a rendered video can be generated from the second unrendered video and second rendering parameters. The display effect of the rendered video matches the first recording mode to a greater extent than the display effect of the preview video matches the first recording mode.

[0109] In some embodiments, the rendering module can also be used to obtain a rendered video based on the first unrendered video and the first rendering parameters after the electronic device saves the first unrendered video and the first rendering parameters, provided that the electronic device meets the second preset conditions; the degree of matching between the display effect of the rendered video and the first recording mode is greater than the degree of matching between the display effect of the preview video and the first recording mode.

[0110] In some embodiments, the rendering module can also be used to optimize the first unrendered video and the first rendering parameters after saving the first unrendered video and the first rendering parameters, provided that the electronic device meets the first preset conditions, to obtain a second unrendered video and the second unrendered parameters; the video quality of the second unrendered video is higher than that of the first unrendered video; provided that the electronic device meets the second preset conditions, a rendered video is obtained based on the second unrendered video and the second rendering parameters; the degree of matching between the display effect of the rendered video and the first recording mode is greater than the degree of matching between the display effect of the preview video and the first recording mode.

[0111] The kernel layer is the layer between hardware and software. The kernel layer contains at least the display driver, camera driver, audio driver, and sensor driver.

[0112] based on Figure 2 The electronic device shown implements the video rendering method in this application embodiment, which can optimize the first unrendered video and the first rendering parameters used to generate the preview video to obtain a second unrendered video with higher video quality and optimized second rendering parameters. Through the second unrendered video and the second rendering parameters, a rendered video with a better display effect than the preview video can be generated. This does not affect the real-time display of the preview video, but also improves the video quality and display effect of the rendered video, thereby improving the user experience.

[0113] The aforementioned electronic devices can be mobile phones, tablets, wearable devices (such as smartwatches, smart bracelets, etc.), smart home devices (such as televisions, etc.), in-vehicle systems (such as in-vehicle computers), etc. Among them, mobile phones can be foldable screen phones or non-foldable screen phones.

[0114] The following uses a mobile phone as an example to illustrate the video rendering method provided in the embodiments of this application.

[0115] See Figure 3 As shown, the above video rendering method may include the following steps S301-S303.

[0116] S301. When the phone is in the first recording mode, the phone acquires the first unrendered video and the first rendering parameters.

[0117] For example, to meet different shooting scenarios and needs, mobile phones offer a variety of recording modes, including Movie Mode, Sports Mode, Night Mode, Portrait Mode, Slow Motion Mode, Time-lapse Mode, Panorama Mode, High Dynamic Range (HDR) Mode, Standard Mode, and Professional Mode. For instance, Movie Mode can achieve an automatic background blur effect, making the subject stand out more and enhancing the sense of depth and three-dimensionality of the image. It also requires high-resolution, high-frame-rate, and other high-quality video recording to make the captured video clearer and smoother. Sports Mode requires video stabilization to reduce blur caused by motion, thus achieving a certain degree of visual stability. Standard Mode provides a balanced color and exposure display effect, suitable for everyday shooting. Professional Mode offers customization options, allowing users to manually adjust parameters such as white balance, ISO, shutter speed, and focal length.

[0118] For example, the first recording mode can be the default recording mode of the phone when recording video, or it can be a recording mode selected by the user from multiple recording modes provided by the phone.

[0119] In some embodiments, the mobile phone first acquires at least one video frame image recorded by the mobile phone in a first recording mode, then the mobile phone acquires an unrendered image of each video frame image after image processing, and finally the mobile phone acquires a first unrendered video based on the unrendered image.

[0120] For example, such as Figure 4 As shown, the user opens the camera app on their phone. The camera app interface offers multiple recording modes. The user can select movie mode (the first recording mode) and click the start button to begin recording video. The phone's camera captures signals from the shooting scene, obtaining data such as... Figure 4The video frame images (raw data) captured by the camera shown are processed by ISP to obtain the unrendered image corresponding to each video frame image. Each unrendered image is the image obtained after the corresponding video frame image has been processed by ISP. After the user clicks the end button to end the recording, all the unrendered images corresponding to the recorded video can be obtained, thus obtaining the first unrendered video.

[0121] For example, the first unrendered video is a video obtained by the mobile phone through ISP processing at the recording end. It does not need to be displayed to the user, and the user is unaware of the first unrendered video.

[0122] As can be seen, the first unrendered video is the recorded video after being processed by the ISP, which can improve the video quality of the subsequently generated preview video.

[0123] In some embodiments, the mobile phone obtains the recording parameters corresponding to the first recording mode, and then, based on the recording parameters corresponding to the first recording mode, the mobile phone obtains the first rendering parameters corresponding to the first recording mode.

[0124] The first rendering parameter is determined according to the first recording mode.

[0125] For example, the first recording mode corresponds to at least one rendering task. Each rendering task is used to add any one of the following display effects to the video processed by the ISP: video stabilization, video blurring, audio enhancement, filter mode, or watermark mode corresponding to the first recording mode. Each rendering task corresponds to one display effect. To achieve the display effect corresponding to each rendering task, each rendering task corresponds to at least one algorithm for calculating first rendering parameters. The calculation algorithms include artificial intelligence (AI) algorithms and traditional algorithms. The first rendering parameters include any one or more of the following parameters: registration information and stabilization trajectory for video stabilization; depth map and blurring trajectory for video blurring; original audio signal for audio enhancement; and color lookup table (LUT) parameters for filter mode. For example, when the first recording mode is movie mode, the rendering tasks corresponding to movie mode are background blurring task and video stabilization task. The calculation algorithm for the first rendering parameters corresponding to the background blurring task includes the calculation algorithm for the depth map of video blurring and the calculation algorithm for the blurring trajectory, etc. The calculation algorithm for the first rendering parameters corresponding to the video stabilization task includes the calculation algorithm for registration information and the calculation algorithm for the stabilization trajectory, etc.

[0126] The recording parameters include image data of each unrendered image in the first unrendered video and camera parameters of the phone when recording in the first recording mode.

[0127] For example, image data can be image feature points, motion parameters, and jitter information in unrendered images. Image feature points can be points in unrendered images that are easy to identify and can be stably tracked between different unrendered images, or points that help identify the foreground and background for constructing a depth map, such as corner points, edges, textures, color contrast, etc. Motion parameters are used to represent the direction and speed of motion of image feature points between adjacent unrendered images in the first unrendered video, reflecting the motion state of the phone, such as parameters of translation, rotation, etc. Alternatively, motion parameters can also be used to represent parameters of relative motion between the foreground and background, such as distinguishing whether the foreground and background are static or dynamic. Jitter information is the jitter trajectory in the first unrendered video identified by comparing the motion parameters between adjacent unrendered images in the first unrendered video, so as to eliminate or reduce these jitter trajectories to achieve the purpose of video stabilization.

[0128] For example, camera parameters include calibration parameters of the mobile phone camera, initialization parameters of the camera, sensor parameters, and rendering parameters. The calibration parameters of the camera include the center position of the camera, focal length, distortion coefficient, and aperture. The initialization parameters of the camera include the initial position of the mobile phone camera and the rotation angle in three coordinate directions. The sensor parameters include the resolution, frame rate, dynamic range (the range of light from the darkest to the brightest that the sensor can capture), and sensitivity (international organization for standardization, ISO) of the camera sensor on the mobile phone. The rendering parameters include the stabilization intensity, smoothness, and bokeh intensity corresponding to the first recording mode.

[0129] For example, the mobile phone can substitute the recording parameters obtained at the recording end into the calculation algorithm of the corresponding first rendering parameter to obtain the corresponding first rendering parameter.

[0130] As can be seen, by recording parameters, the mobile phone can obtain first rendering parameters that are more consistent with the first recording mode on the mobile phone recording end, thereby improving the display effect of the subsequently generated preview video.

[0131] The first unrendered video and the first rendering parameters obtained by the mobile phone at the recording end are used to generate a preview video of the first unrendered video.

[0132] For example, such as Figure 5As shown, the mobile phone acquires multiple video frame images captured by the camera, performs ISP image signal processing on each video frame image to obtain a first unrendered video, saves the first unrendered video in the phone's memory, obtains the calculation algorithm of the first rendering parameters corresponding to the first recording mode, and obtains the first rendering parameters based on the first unrendered video and the calculation algorithm of the first rendering parameters, saves the first rendering parameters in the phone's memory, and calls the preview rendering algorithm corresponding to each of the at least one rendering task corresponding to the first recording mode, and can add the display effect corresponding to the rendering task to the first unrendered video based on the first rendering parameters to generate a preview video and display it to the user.

[0133] The preview rendering algorithm can be either an AI algorithm or a traditional algorithm. The algorithm must meet the conditions of real-time preview video display and the phone's power consumption not exceeding a preset limit. Each rendering task corresponds to one of the following display effects: video stabilization, video blurring, audio enhancement, filter mode, or watermark mode. Due to the constraints of real-time display and phone power consumption, the generated preview videos are of low quality, and their display effects cannot meet users' demands for high-quality videos.

[0134] S302, The phone saves the first unrendered video and the first rendered parameters.

[0135] For example, the mobile phone encodes and encapsulates the first unrendered video obtained in the first recording mode into a video stream and stores it. The mobile phone also encodes and encapsulates the first rendering parameters obtained in the first recording mode into a bitstream and stores it. The video stream of the first unrendered video and the bitstream of the first rendering parameters can be stored in the internal memory of the mobile phone or in the external memory of the mobile phone.

[0136] For example, the first unrendered video stored in the phone can display the first frame of the first unrendered video as a thumbnail in the phone's gallery. When the user clicks on the first unrendered video in the phone's gallery to view it, the phone displays the video after rendering it according to the display effect corresponding to the first recording mode. The phone's "rendering and playing at the same time" function enables the user to view it without being aware of it, thus improving the user experience.

[0137] S303. When the mobile phone meets the first preset conditions, the first unrendered video and the first rendering parameters are processed to obtain the second unrendered video and the second rendering parameters.

[0138] For example, the first preset condition includes any one of the following: the user triggers a control to render the first unrendered video, the user triggers a control to edit the first unrendered video, or the mobile phone is in an idle state.

[0139] For example, if the phone meets the first preset condition, the phone can use a more complex algorithm on the gallery side to optimize the first unrendered video and the first rendering parameters to obtain a second unrendered video and second rendering parameters. The second unrendered video obtained by the phone does not need to be displayed to the user. The phone saves the second unrendered video and the second rendering parameters in the phone's memory. The first unrendered video stored in the memory can be replaced with the second unrendered video, and the first rendering parameters can be replaced with the second rendering parameters. The phone can display the first frame of the stored second unrendered video in the phone's gallery as a thumbnail. When the user clicks to view the second unrendered video in the gallery, the phone displays the video after rendering the second unrendered video according to the display effect corresponding to the first recording mode. The phone's "rendering and playing at the same time" function realizes the user's seamless viewing and improves the user experience.

[0140] In some embodiments, when a mobile phone obtains a second unrendered video, the mobile phone first obtains at least one unrendered image of the first unrendered video, and then processes the at least one unrendered image to obtain the second unrendered video.

[0141] For example, the mobile phone unpacks and decodes the video stream of the first unrendered video stored in the mobile phone. When processing the first unrendered video, the mobile phone can perform frame-by-frame optimization of the first unrendered video. That is, it performs image optimization processing on each unrendered image in the first unrendered video. The algorithm used for image optimization processing includes at least one of the following algorithms for optimizing the first unrendered video: noise reduction, detail enhancement, deblurring, super-resolution, frame interpolation, audio enhancement, etc. The algorithm used for image optimization processing can be an AI algorithm or a traditional algorithm. The algorithm used for image optimization processing is more complex than the algorithm used for ISP processing mentioned above. The resulting second unrendered video has a higher video quality than the first unrendered video.

[0142] For example, video quality is determined based on at least one of resolution, frame rate, bit rate, compression ratio, distortion, and noise. The video quality of a first unrendered video can be determined by at least one of resolution, frame rate, bit rate, compression ratio, distortion, and noise in a first unrendered video, and the video quality of a second unrendered video can be determined by at least one of resolution, frame rate, bit rate, compression ratio, distortion, and noise in a second unrendered video, thereby determining that the video quality of the second unrendered video is higher than that of the first unrendered video.

[0143] For example, the mobile phone can call the image optimization processing algorithm corresponding to each optimization task according to the optimization task corresponding to the first recording mode. When the first recording mode is movie mode, the optimization tasks corresponding to movie mode are noise reduction and de-shaking. After the mobile phone unpacks and decodes the video stream of the stored first unrendered video, it performs noise reduction optimization processing on the first unrendered video according to the image optimization processing algorithm corresponding to noise reduction, and performs de-shaking optimization processing on the first unrendered video according to the image optimization processing algorithm corresponding to de-shaking, to obtain the second unrendered video.

[0144] For example, after obtaining the second unrendered video, the mobile phone can encode and packetize the second unrendered video to obtain the video stream of the second unrendered video, and store the video stream of the second unrendered video. When storing, the mobile phone can replace the video stream of the first unrendered video with the video stream of the second unrendered video; it can also store the video stream of the second unrendered video and delete the video stream of the first unrendered video; or it can store both the video stream of the first unrendered video and the video stream of the second unrendered video.

[0145] As can be seen, the mobile phone performs image optimization processing frame by frame on the first unrendered video to obtain the second unrendered video, thereby improving the video quality of the second unrendered video.

[0146] In some embodiments, the mobile phone can call the rendering parameter optimization processing algorithm corresponding to each rendering task according to the rendering task corresponding to the first recording mode, and process the first rendering parameters according to the rendering parameter optimization processing algorithm corresponding to each rendering task to obtain the second rendering parameters.

[0147] For example, the mobile phone unpacks and decodes the bitstream of the first rendering parameters stored in the phone. When processing the first rendering parameters, it calls the algorithm for optimizing each rendering parameter corresponding to the first recording mode. The algorithm for optimizing the rendering parameters includes at least one of the following algorithms: depth map super-resolution, focus tracking, image stabilization trajectory optimization, and filter parameter optimization. For example, the algorithm for optimizing the rendering parameters can be an AI algorithm or a traditional algorithm. The algorithm for optimizing the rendering parameters is more complex than the calculation algorithm for the first rendering parameters, and the display effect of the video rendered by the obtained second rendering parameters is better than the display effect of the video rendered by the first rendering parameters.

[0148] For example, when the first recording mode is movie mode, the rendering tasks corresponding to movie mode are video stabilization and video blurring. After storing the second unrendered video, the phone optimizes the first rendering parameters. It can also optimize the first rendering parameters again when the phone meets the first preset condition. First, the phone depackets and decodes the bitstream of the first rendering parameters stored in the phone. Based on the algorithm for optimizing the rendering parameters corresponding to video stabilization and the second unrendered video, the phone performs stabilization optimization on the first rendering parameters. Based on the algorithm for optimizing the rendering parameters corresponding to video blurring and the second unrendered video, the phone performs blurring optimization on the first rendering parameters to obtain the second rendering parameters.

[0149] For example, when the mobile phone meets the first preset condition, it can also simultaneously optimize the first unrendered video and the first rendering parameters to obtain the second unrendered video and the second rendering parameters. When performing optimization simultaneously, the mobile phone can optimize the first unrendered video using an image optimization algorithm to obtain the second unrendered video. The mobile phone can also optimize the first rendering parameters using a rendering parameter optimization algorithm and the first unrendered video to obtain the second rendering parameters. In this way, the mobile phone can obtain the second unrendered video and the second rendering parameters simultaneously. Alternatively, the mobile phone can obtain the second unrendered video and the second rendering parameters step by step. After obtaining the second unrendered video, the mobile phone can optimize the first rendering parameters using a rendering parameter optimization algorithm and the second unrendered video to obtain the second rendering parameters.

[0150] It is evident that the mobile phone optimizes the first rendering parameter using a rendering parameter optimization algorithm to obtain the second rendering parameter, thereby improving the rendering effect of the second rendering parameter.

[0151] In some embodiments, when the mobile phone obtains the second rendering parameters, it first obtains the editing parameters that the user edits on the first unrendered video, and then processes the first rendering parameters according to the editing parameters to obtain the second rendering parameters.

[0152] For example, the mobile phone can also respond to the user's editing operation on the mobile phone screen, obtain editing parameters, and the mobile phone can optimize the first rendering parameters based on the editing parameters and the rendering parameter optimization algorithm, for example, such as... Figure 6As shown, when the first recording mode is movie mode, under the first preset condition, the first unrendered video is processed by an optimization algorithm (image optimization processing) for noise reduction or shaking removal to obtain a second unrendered video. The first unrendered video stored in the phone's memory is then replaced with the second unrendered video. When the phone again meets the first preset condition, the second unrendered video, the first rendering parameters, and the user's editing parameters are retrieved from the memory. The editing parameters include stabilization strength, stabilization trajectory, bokeh strength, and focus, etc. The editing parameters can be selected by the user by dragging a slider or by the user entering them into an input box. The settings can be selected by the user through a drop-down menu or by clicking on the phone's editing interface. For example, the stabilization strength is indicated by a slider on the phone screen, allowing the user to select the appropriate stabilization level. Similarly, the bokeh strength is indicated by a drop-down menu, allowing the user to select a suitable bokeh level. Users can also select the focus of the second unrendered video by clicking on the editing interface displayed on the phone screen, or by entering coordinate data in the input box to determine the stabilization trajectory. After obtaining the editing parameters, the phone can call the rendering parameter optimization algorithm corresponding to the first recording mode. The second unrendered video, the first rendering parameters, and the editing parameters are input into the corresponding rendering parameter optimization algorithm to obtain the second rendering parameters. The first rendering parameters stored in the phone's memory are then replaced with the second rendering parameters. The algorithms for optimizing rendering parameters in movie mode can include electronic image stabilization trajectory planning algorithms, depth map optimization algorithms, and saliency detection and tracking algorithms. The phone can utilize the electronic image stabilization trajectory planning algorithm, combined with the stabilization intensity and trajectory adjustment information input by the user through the editing interface, and utilize the depth map optimization algorithm and saliency detection and tracking algorithm, combined with the focus point and bokeh intensity adjustment information input by the user through the editing interface, to generate better second rendering parameters. This achieves more advanced stabilization and bokeh effects such as "gimbal-like" camera movement, "sliding-like" camera movement, and hair-level segmentation, which better meet the user's needs.

[0153] As can be seen, mobile phones can optimize the first rendering parameters through user editing, meeting the different users' needs for video rendering and improving the user experience.

[0154] For example, when the phone is optimizing the first unrendered video and the first rendering parameters, if the user operates the phone, such as listening to music or playing games, which consumes phone resources, the phone will stop optimizing the first unrendered video and the first rendering parameters. When the phone meets the first preset condition again, the phone will resume optimizing the first unrendered video and the first rendering parameters, so that the phone's performance is not affected when the user operates the phone, thus ensuring the user's operating experience.

[0155] For example, after the mobile phone obtains the second rendering parameters, it encodes and packages the second rendering parameters to obtain the bitstream of the second rendering parameters. The bitstream of the second rendering parameters is stored. When storing, the mobile phone can replace the bitstream of the first rendering parameters with the bitstream of the second rendering parameters, or it can store the bitstream of the second rendering parameters and delete the bitstream of the first rendering parameters, or it can store both the bitstream of the first rendering parameters and the bitstream of the second rendering parameters.

[0156] For example, such as Figure 7 As shown, when the mobile phone meets the first preset condition, it performs data preprocessing on the first unrendered video and the first rendering parameters stored in the mobile phone. For example, the mobile phone unpacks and decodes the video stream of the first unrendered video to obtain the first unrendered video, and unpacks and decodes the bitstream of the first rendering parameters to obtain the first rendering parameters. Then, the mobile phone optimizes the unpacked and decoded first unrendered video to obtain the second unrendered video, and optimizes the first rendering parameters to obtain the second rendering parameters. Next, the mobile phone encodes the second unrendered video to obtain the video stream of the second unrendered video, and encodes the second rendering parameters to obtain the bitstream of the second rendering parameters. Finally, the mobile phone performs data replacement, replacing the video stream of the first unrendered video stored in the mobile phone's memory with the video stream of the second unrendered video, and replacing the bitstream of the first rendering parameters with the bitstream of the second rendering parameters. The video stream of the second unrendered video and the bitstream of the second rendering parameters stored in the mobile phone are used to generate a rendered video, which the user can view / edit / share / upload.

[0157] The steps S301-S303 described above implement the video rendering method in this application embodiment. It can save the first unrendered video and the first rendering parameters used to generate the preview video. Then, under the condition that the mobile phone meets the first preset condition, the saved first unrendered video and the first rendering parameters are optimized to obtain a second unrendered video with higher video quality and optimized second rendering parameters. With the second unrendered video and the second rendering parameters, a rendered video with a better display effect than the preview video can be generated. This does not affect the real-time display of the preview video, but also improves the video quality and display effect of the rendered video, thereby improving the user experience.

[0158] In some embodiments, when the mobile phone meets the second preset conditions, the mobile phone obtains a rendered video based on the second unrendered video and the second rendering parameters.

[0159] For example, the second preset condition includes any one of the following: the user views the second unrendered video, the user shares the second unrendered video, the user uploads the second unrendered video, the user triggers a control to render the second unrendered video, the user triggers a control to edit the second unrendered video, or the mobile phone is in an idle state.

[0160] For example, when the phone is idle, a first preset condition can be triggered. The phone can then use a more complex algorithm on the gallery side to optimize the first unrendered video and the first rendering parameters. After obtaining the second unrendered video and the second rendering parameters, if the phone is still idle, it can use a more complex algorithm on the gallery side to generate a rendered video based on the second unrendered video and the second rendering parameters. The phone can save the generated rendered video in its storage and display the first frame of the rendered video as a thumbnail in the phone's gallery, allowing users to directly view / upload / share the rendered video, further improving the efficiency of using rendered videos.

[0161] It is evident that by optimizing the second unrendered video and the second rendering parameters, the mobile phone can obtain a rendered video with higher quality and better display effect. The display effect of the rendered video is more in line with the display requirements of the first recording mode than the display effect of the preview video, thereby improving the user's demand for high-quality rendered video and thus enhancing the user experience.

[0162] In some embodiments, when a mobile phone obtains a rendered video, it first acquires at least one video frame image of a second unrendered video, and then renders the at least one video frame image according to the second rendering parameters to obtain the rendered video.

[0163] For example, such as Figure 8 As shown, after the mobile phone detects that the second preset condition is met, it unpacks and decodes the video stream of the stored second unrendered video and the bitstream of the second rendering parameters to obtain the second unrendered video and the second rendering parameters. According to at least one rendering task corresponding to the first recording mode, it calls the rendering algorithm corresponding to each rendering task, and inputs the decoded second unrendered video and the second rendering parameters into the corresponding rendering algorithm. The rendering algorithm can render each video frame of the second unrendered video frame by frame through the second rendering parameters to add the display effect corresponding to the rendering task to the second unrendered video, thereby obtaining the rendered video. The rendered video is then encoded to obtain the video stream of the rendered video, and the video stream of the rendered video is stored in the memory of the mobile phone. The rendered video has the display effect of each rendering task corresponding to the first recording mode.

[0164] For example, the rendering algorithm corresponding to each rendering task includes any one of the following algorithms: video stabilization, video blurring, audio enhancement, filter mode, watermarking, etc. The rendering algorithm can be an AI algorithm or a traditional algorithm. The rendering algorithm is not constrained by the real-time performance and power consumption of the preview when it is used. Therefore, the rendering algorithm used is more complex than the preview rendering algorithm used when generating the preview video. Thus, the display effect of the video rendered by the rendering algorithm is better than the rendering effect of the video rendered by the preview rendering algorithm. In other words, the display effect of the rendered video matches the first recording mode better than the display effect of the preview video matches the first recording mode.

[0165] The display effects include at least one of the following: video stabilization, video blurring, audio enhancement, filter mode, and watermark mode.

[0166] For example, such as Figure 9 As shown, when a user views the second rendered video, the phone meets the second preset condition, retrieves the video stream of the second unrendered video and the bitstream of the second rendering parameters from the phone's memory, unpacks and decodes the video stream of the second unrendered video and the bitstream of the second rendering parameters to obtain the second unrendered video and the second rendering parameters. Starting from the first frame of the second unrendered video, the phone calls the rendering algorithm to render the second unrendered video frame by frame, and can display the video frame images of the rendered second unrendered video in real time, realizing the phone's "rendering and playing at the same time" function, thereby enabling the user to view it without noticing, until the last frame of the second unrendered video is rendered, obtaining the complete rendered video. After displaying the rendered video frame image of the last frame of the second unrendered video to the user, the rendered video playback ends, the rendered video is encoded into data to obtain the video stream of the rendered video, and the video stream of the rendered video is stored in the phone's memory so that the rendered video can be played directly when the user views it in the phone's gallery next time.

[0167] For example, a mobile phone can encode and encapsulate the rendered video to obtain the video stream of the rendered video. The mobile phone can store the video stream of the rendered video so that it can be played directly when viewed in the gallery next time.

[0168] For example, when the first recording mode is movie mode, the rendering algorithm includes a stabilization algorithm, a bokeh algorithm, and a watermarking algorithm. When the phone performs the video stabilization rendering task, it calls the stabilization algorithm; when it performs the video bokeh rendering task, it calls the background blurring algorithm and the watermarking algorithm. The phone generates a rendered video with stabilization, bokeh, and watermark effects in real time. When the user views the photos in the gallery, the phone will display the rendered video generated in real time. The user will not be aware of the execution of the rendering algorithm. When the phone is idle, it will generate and store the rendered video. When the user wants to share the video, if the rendered video has not yet been generated, the phone can remind the user to wait for the rendered video to be generated before transmitting and sharing.

[0169] As can be seen, when the phone shoots in the first recording mode, it does not store the preview video. When viewing the second unrendered video in the gallery or editing the first unrendered video, the optimized second unrendered video and the second rendering parameters are used to employ a more complex rendering algorithm to render the video frame by frame, resulting in a rendered video with better display effects and more in line with the user's personalized needs, thus improving the user experience.

[0170] In some embodiments, the mobile phone may perform unrendered video and rendering parameter optimization multiple times for generating rendered video; in other embodiments, the mobile phone may not perform unrendered video and rendering parameter optimization.

[0171] For example, when the phone is in the first recording mode, the phone acquires the first unrendered video and the first rendering parameters, saves the first unrendered video and the first rendering parameters, and when the phone meets the second preset conditions, the phone obtains the rendered video based on the first unrendered video and the first rendering parameters.

[0172] For example, when a user views a photo in the gallery, if the first unrendered video and the first rendering parameters have not yet been optimized, the phone can generate and display a rendered video in real time based on the first unrendered video and the first rendering parameters. The rendering algorithm used to generate the rendered video is more complex than the preview rendering algorithm used to generate the preview video. Even without optimizing the first unrendered video and the first rendering parameters, the display effect of the generated rendered video is still better than that of the preview video.

[0173] For example, when the mobile phone meets the second preset conditions, and obtains the rendered video based on the first unrendered video and the first rendering parameters, firstly, the mobile phone can process the first unrendered video to obtain the second unrendered video when the electronic device meets the first preset conditions. Then, the mobile phone can update the first unrendered video based on the second unrendered video. Finally, when the mobile phone meets the second preset conditions, it obtains the rendered video based on the updated first unrendered video and the first rendering parameters.

[0174] For example, when the mobile phone meets the second preset conditions, and when obtaining the rendered video based on the updated first unrendered video and the first rendering parameters, firstly, when the mobile phone meets the first preset conditions, the first rendering parameters are optimized to obtain the second rendering parameters. Then, the mobile phone updates the first rendering parameters based on the second rendering parameters. Finally, when the mobile phone meets the second preset conditions, the mobile phone obtains the rendered video based on the updated first unrendered video and the updated first rendering parameters.

[0175] For example, a rendered video generated based on a first unrendered video and first rendering parameters is stored. When the user views it next time, if the first unrendered video has not yet been optimized, the stored rendered video can be played directly. If the first unrendered video has been optimized, the phone receives an optimized second unrendered video. The phone then regenerates and displays the rendered video based on the second unrendered video and the first rendering parameters, replacing the previously stored rendered video with the newly generated rendered video. If the first unrendered video and the first rendering parameters have been optimized, the phone receives an optimized second unrendered video and optimized second rendering parameters. The phone then regenerates and displays the rendered video based on the second unrendered video and the second rendering parameters, replacing the previously stored rendered video with the newly generated rendered video.

[0176] For example, when a mobile phone records video in movie mode, the first unrendered video obtained by the phone at the recording end is as follows: Figure 10 As shown, the first unrendered video does not have a blurred display effect. The first rendering parameters obtained by the phone at the recording end are as follows: Figure 11 As shown, the first rendering parameter is a depth map. Under the condition that the phone meets the first preset condition, the phone optimizes the first unrendered video and the first rendering parameter on the gallery side, and the resulting optimized second unrendered video is shown below. Figure 12 As shown, the second unrendered video is the first unrendered video after noise reduction processing, but the blurring effect has not yet been rendered. The optimized second rendering parameters are as follows. Figure 13 As shown, the second rendering parameter is the depth map after depth map optimization processing of the first rendering parameter. Under the second preset condition, the phone uses a rendering algorithm on the gallery side to render the second unrendered video using the second rendering parameter, and the resulting rendered video is as follows. Figure 14 As shown, this is a rendered video with a blurred display effect.

[0177] As can be seen, when the recording ends, the first unrendered video stored in the phone is not rendered for display. After the phone meets the second preset condition, it renders the first unrendered video to obtain a rendered video, improving the display effect of the rendered video. After the phone meets the first preset condition again, it optimizes the first unrendered video and the first rendering parameters to generate a second unrendered video and second rendering parameters, updating the first unrendered video stored in the phone to the second unrendered video, thus improving the video quality. After the phone meets the second preset condition again, it renders the second unrendered video using the second rendering parameters to generate a rendered video. The phone replaces the previously generated rendered video with this rendered video, so that the rendered video with higher quality and better display effect can be viewed / uploaded / shared in the gallery, while taking into account both the real-time preview of the video display and the high-quality rendered video.

[0178] It is understood that, in order to achieve the above functions, the aforementioned electronic device includes hardware and / or software modules corresponding to perform each function. Based on the algorithmic steps of the various examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in conjunction with the embodiments, but such implementation should not be considered beyond the scope of this application.

[0179] This application embodiment can divide the electronic device into functional modules according to the above method example. For example, each function can be divided into its own functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware. It should be noted that the module division in this embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0180] This application also provides an electronic device, such as... Figure 15 As shown, the electronic device may include one or more processors 1501, memory 1502 and communication interface 1503.

[0181] The memory 1502, communication interface 1503, and processor 1501 are coupled together. For example, the memory 1502, communication interface 1503, and processor 1501 can be coupled together via bus 1504.

[0182] The communication interface 1503 is used for data transmission with other devices. The memory 1502 stores computer program code. The computer program code includes computer instructions, which, when executed by the processor 1501, cause the electronic device to perform the video rendering method described in this embodiment.

[0183] The processor 1501 may be a processor or controller, such as a central processing unit (CPU), a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in connection with this disclosure. The processor may also be a combination that implements computational functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0184] Bus 1504 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Bus 1504 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 15 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0185] This application also provides a computer-readable storage medium storing computer program code. When the processor executes the computer program code, the electronic device executes the relevant method steps in the above method embodiments.

[0186] The electronic devices and computer storage media provided in this application are used to execute the corresponding methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects of the corresponding methods provided above, and will not be repeated here.

[0187] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0188] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0189] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0190] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0191] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, in essence, or the part that contributes, or all or part of the technical solution, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0192] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A video rendering method, characterized in that, Applied to electronic devices, including: When the electronic device is in a first recording mode, a first unrendered video and first rendering parameters are acquired; wherein, the first unrendered video and the first rendering parameters are used to generate a preview video of the first unrendered video; the first rendering parameters are determined according to the first recording mode; Save the first unrendered video and the first rendering parameters; When the electronic device meets the first preset condition, the first unrendered video and the first rendering parameters are processed to obtain a second unrendered video and a second rendering parameter; the second unrendered video and the second rendering parameter are used to obtain a rendered video.

2. The method according to claim 1, characterized in that, The method further includes: When the electronic device meets the second preset conditions, a rendered video is obtained based on the second unrendered video and the second rendering parameters.

3. The method according to claim 2, characterized in that, The second preset condition includes any one of the following: a user viewing a second unrendered video, a user sharing a second unrendered video, a user uploading a second unrendered video, a user triggering a control to render the second unrendered video, a user triggering a control to edit the second unrendered video, or the electronic device being in an idle state.

4. The method according to any one of claims 1-3, characterized in that, The first preset condition includes any one of the following: the user triggers a control to render the first unrendered video, the user triggers a control to edit the first unrendered video, or the electronic device is in an idle state.

5. The method according to claim 4, characterized in that, Obtaining the second rendering parameters includes: Obtain the editing parameters that the user edits on the first unrendered video; The first rendering parameters are processed according to the editing parameters to obtain the second rendering parameters.

6. The method according to any one of claims 1-5, characterized in that, The process of obtaining the first unrendered video includes: Acquire at least one video frame image recorded by the electronic device in the first recording mode; Obtain the unrendered image of each video frame image after image processing from the at least one video frame image; Based on the unrendered image, obtain the first unrendered video.

7. The method according to any one of claims 1-6, characterized in that, The process of obtaining the first rendering parameters includes: Obtain the recording parameters corresponding to the electronic device in the first recording mode; the recording parameters include the image data of each unrendered image in the first unrendered video and the camera parameters of the electronic device when recording in the first recording mode. Based on the recording parameters corresponding to the first recording mode, obtain the first rendering parameters corresponding to the first recording mode.

8. The method according to claim 6 or 7, characterized in that, The process of obtaining the second unrendered video includes: Obtain at least one unrendered image from the first unrendered video; The at least one unrendered image is processed to obtain a second unrendered video.

9. The method according to claim 2 or 3, characterized in that, The obtained rendered video includes: Obtain at least one video frame image of the second unrendered video; The at least one video frame image is rendered according to the second rendering parameters to obtain a rendered video.

10. A video rendering method, characterized in that, Applied to electronic devices, including: When the electronic device is in a first recording mode, a first unrendered video and first rendering parameters are acquired; the first unrendered video and the first rendering parameters are used to generate a preview video of the first unrendered video; the first rendering parameters are determined according to the first recording mode. Save the first unrendered video and the first rendering parameters; When the electronic device meets the second preset conditions, a rendered video is obtained based on the first unrendered video and the first rendering parameters.

11. The method according to claim 10, characterized in that, When the electronic device meets the second preset condition, obtaining the rendered video based on the first unrendered video and the first rendering parameters includes: When the electronic device meets the first preset condition, the first unrendered video is processed to obtain the second unrendered video; Update the first unrendered video based on the second unrendered video; When the electronic device meets the second preset conditions, a rendered video is obtained based on the updated first unrendered video and the first rendering parameters.

12. The method according to claim 11, characterized in that, When the electronic device meets the second preset condition, obtaining the rendered video based on the updated first unrendered video and the first rendering parameters includes: When the electronic device meets the first preset condition, the first rendering parameters are processed to obtain the second rendering parameters; Update the first rendering parameters according to the second rendering parameters; When the electronic device meets the second preset conditions, a rendered video is obtained based on the updated first unrendered video and the updated first rendering parameters.

13. An electronic device, characterized in that, The device includes a memory and one or more processors; the memory is coupled to the processors; the memory stores computer program code, the computer program code including computer instructions, which, when executed by the processor, cause the electronic device to perform the video rendering method as described in any one of claims 1-9, or to perform the video rendering method as described in any one of claims 10-12.

14. A computer-readable storage medium, characterized in that, The method includes computer instructions that, when executed on an electronic device, cause the electronic device to perform the video rendering method as described in any one of claims 1-9, or to perform the video rendering method as described in any one of claims 10-12.

15. A computer program product, characterized in that, When the computer program product is run on an electronic device, it causes the electronic device to perform the video rendering method as described in any one of claims 1-9, or to perform the video rendering method as described in any one of claims 10-12.

Images