Video processing method and device, electronic equipment and storage medium
By using multi-chip collaborative processing and identification management in electronic devices, the problem of not adding effects to some layers in high-dynamic range videos is solved, ensuring the color effect during video playback and improving user experience.
Patent Information
- Application Number
- CN202410077158.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-01-18
- Publication Date
- 2025-07-18
AI Technical Summary
When processing high dynamic range videos in the prior art, some layers fail to add high dynamic range effects, resulting in obvious color difference during video playback, affecting the user's viewing experience.
By using multiple chips in electronic devices to process multiple initial layers of video source data, including a central processor, graphics processor and image enhancement chip, respectively or jointly perform image processing and merging, setting up pending identifications to ensure that layers without adding high dynamic range effects are performed after synthesis.
It effectively avoids the video loss of high dynamic range effect, ensures the color effect during video playback, and improves the user's viewing experience.
Smart Images
Figure CN120343420A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of electronic devices, and particularly to a video processing method, apparatus, electronic device, and storage medium. Background Art
[0002] In recent years, with the rapid development of image processing technology, High Dynamic Range (HDR) technology has become increasingly mature and widely used. HDR has better effects in aspects such as brightness, color gamut, and color depth, making HDR videos provide a better viewing experience. When playing an HDR video, it is usually necessary to perform special image processing on layers through a main control chip to make the video have an HDR effect.
[0003] However, when using the video processing method of related technologies to process HDR videos, in some scenarios, due to the processing power limitation of the main control chip, some layers that need to add the HDR effect are not subjected to special image processing, resulting in the loss of the HDR effect in the video and problems such as obvious color differences during video playback, which affect the user's viewing experience. Summary of the Invention
[0004] To overcome the problems in related technologies, the present disclosure provides a video processing method, apparatus, electronic device, and storage medium.
[0005] According to the first aspect of the embodiments of the present disclosure, there is provided a video processing method applied to an electronic device. The video processing method includes:
[0006] When the electronic device plays a video, perform image processing and merging on multiple initial layers of a frame image in video source data to obtain a composite layer, where the multiple initial layers include a first initial layer that needs to add an HDR effect;
[0007] If there is the first initial layer that has not added the HDR effect after the image processing, perform a first effect addition process on the composite layer to obtain a target layer, where the first effect addition process is used to add the HDR effect to the composite layer;
[0008] Display the target layer.
[0009] In some embodiments of the present disclosure, the performing image processing and merging on each initial layer of the frame image in the video source data to obtain a composite layer includes:
[0010] Perform image processing on some of the multiple initial layers through a first chip respectively to obtain at least one first intermediate layer;
[0011] The second chip performs image processing and merging on another part of the plurality of initial layers to obtain a second intermediate layer;
[0012] The at least one first intermediate layer and the second intermediate layer are merged to obtain the composite layer;
[0013] Wherein, when the first initial layer is included in the initial layers processed by the first chip, the first chip performs a second effect addition process on the first initial layer, and the second effect addition process is used to add the high dynamic range effect to the first initial layer.
[0014] In some embodiments of the present disclosure, before performing image processing and merging on the plurality of initial layers of the frame images in the video source data to obtain a composite layer, the video processing method further includes:
[0015] Set a to-be-processed flag for the first initial layer among the plurality of initial layers;
[0016] After the first chip performs the second effect addition process on the first initial layer, clear the to-be-processed flag of the first initial layer or modify the to-be-processed flag to a processed flag;
[0017] If the first initial layer is included in the initial layers processed by the second chip, set the to-be-processed flag for the second intermediate layer.
[0018] In some embodiments of the present disclosure, the merging of the at least one first intermediate layer and the second intermediate layer to obtain the composite layer includes:
[0019] If there is a layer with the to-be-processed flag among the at least one first intermediate layer and the second intermediate layer, set the to-be-processed flag for the composite layer.
[0020] In some embodiments of the present disclosure, the performing a first effect addition process on the composite layer to obtain a target layer if there is a first initial layer that has not been added with the high dynamic range effect after the image processing includes:
[0021] If the composite layer has the to-be-processed flag, determine that there is a first initial layer that has not been added with the high dynamic range effect, and perform a first effect addition process on the composite layer to obtain a target layer.
[0022] In some embodiments of the present disclosure, the performing a first effect addition process on the composite layer to obtain a target layer includes:
[0023] Perform linearization processing on the composite layer to obtain a first layer;
[0024] Preprocess the first layer to obtain a second layer;
[0025] Perform filtering on the second layer to obtain a third layer;
[0026] Perform tone curve mapping on the third layer to obtain a fourth layer;
[0027] Perform gamma correction on the fourth layer to obtain the target layer.
[0028] In some embodiments of the present disclosure, the performing filtering on the second layer includes:
[0029] Perform bilateral filtering on the second layer to obtain the third layer.
[0030] In some embodiments of the present disclosure, the performing the first effect addition process on the composite layer to obtain the target layer includes:
[0031] Perform the first effect addition process on the composite layer through a third chip to obtain the target layer.
[0032] In some embodiments of the present disclosure, the first chip includes a central processing unit, the second chip includes a graphics processing unit, and the third chip includes an image enhancement chip.
[0033] According to a second aspect of the embodiments of the present disclosure, there is provided a video processing apparatus, which is applied to an electronic device. The video processing apparatus includes:
[0034] A synthesis module, which is configured to perform image processing and merging on multiple initial layers of frame images in video source data when the electronic device plays a video, to obtain a composite layer. The multiple initial layers include a first initial layer that needs to add a high dynamic range effect;
[0035] An effect addition module, which is configured to, if there is a first initial layer that has not added the high dynamic range effect after the image processing, perform a first effect addition process on the composite layer to obtain a target layer. The first effect addition process is used to add the high dynamic range effect to the composite layer;
[0036] A display module, which is configured to display the target layer.
[0037] According to a third aspect of the embodiments of the present disclosure, there is provided an electronic device, which includes:
[0038] A processor;
[0039] A memory for storing processor-executable instructions;
[0040] Wherein, the processor is configured to:
[0041] When the electronic device plays a video, perform image processing and merging on multiple initial layers of frame images in the video source data to obtain a composite layer, and the multiple initial layers include a first initial layer that needs to add a high dynamic range effect;
[0042] If there is the first initial layer that has not added the high dynamic range effect after the image processing, perform a first effect addition process on the composite layer to obtain a target layer, and the first effect addition process is used to add the high dynamic range effect to the composite layer;
[0043] Display the target layer.
[0044] According to a fourth aspect of the embodiments of the present disclosure, there is provided a non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by a processor of an electronic device, enabling the electronic device to execute a video processing method, and the video processing method includes:
[0045] When the electronic device plays a video, perform image processing and merging on multiple initial layers of frame images in the video source data to obtain a composite layer, and the multiple initial layers include a first initial layer that needs to add a high dynamic range effect;
[0046] If there is the first initial layer that has not added the high dynamic range effect after the image processing, perform a first effect addition process on the composite layer to obtain a target layer, and the first effect addition process is used to add the high dynamic range effect to the composite layer;
[0047] Display the target layer.
[0048] The technical solutions provided by the embodiments of the present disclosure may include the following beneficial effects: In the case where there is a first initial layer that has not added a high dynamic range effect after image processing, by performing a first effect addition process on the composite layer, the obtained target layer has a high dynamic range effect, which can avoid the video losing the original high dynamic range effect, ensure the color effect during the video playback, and improve the user's viewing experience.
[0049] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. Brief Description of the Drawings
[0050] The accompanying drawings here are incorporated into the specification and form a part of this specification, showing embodiments consistent with the present invention, and are used together with the specification to explain the principles of the present invention.
[0051] Figure 1 is a flowchart of a video processing method shown according to an exemplary embodiment.
[0052] Figure 2 is a flowchart of performing image processing and merging on each initial layer of a frame image in video source data to obtain a composite layer shown according to an exemplary embodiment.
[0053] Figure 3 is a flowchart of a video processing method shown according to another exemplary embodiment.
[0054] Figure 4 is a flowchart of performing a first effect addition process on the composite layer to obtain a target layer shown according to an exemplary embodiment.
[0055] Figure 5 is a timing diagram when an electronic device executes a video control method shown according to an exemplary embodiment.
[0056] Figure 6 is a flowchart of a video processing method shown according to another exemplary embodiment.
[0057] Figure 7 is a block diagram of a video processing apparatus shown according to an exemplary embodiment.
[0058] Figure 8 is a block diagram of an electronic device shown according to an exemplary embodiment.
[0059] In the figure:
[0060] 10 - Composite module; 20 - Effect addition module; 30 - Display module; 101 - Processing component; 102 - Memory; 103 - Power component; 104 - Multimedia component; 105 - Audio component; 106 - Input / output interface; 107 - Sensor component; 108 - Communication component; 109 - Processor. Detailed implementation manners
[0061] Here, the exemplary embodiments will be described in detail, and the examples are shown in the accompanying drawings. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with the present invention. On the contrary, they are merely examples of apparatuses and methods consistent with some aspects of the present invention as detailed in the appended claims.
[0062] In recent years, with the rapid development of image processing technology, high dynamic range technology has become increasingly mature and its application scope has become more extensive. As a processing technology to improve image brightness and contrast, high dynamic range can enhance the brightness, color gamut, and details of the picture, making the performance of images and videos closer to the real visual experience, and making high dynamic range videos have a better viewing experience compared to standard dynamic range (SDR) videos.
[0063] In related technologies, high dynamic range videos need to be synthesized by the hardware (Device) of a main control chip such as a central processing unit (CPU) to achieve the high dynamic range effect of the video. The main control chip performs special image processing such as the reverse tone mapping algorithm on some layers of the frame image through the HWComposer module used for window synthesis display in the Device and the hardware abstraction layer, so that the video is mapped to the high dynamic range.
[0064] However, when processing high dynamic range videos using the video processing method of related technologies, when the display screen rotates or the screen is reduced or enlarged by more than a certain range, the main control chip will have insufficient layer processing and layer synthesis capabilities. At least some layers will be switched to other chips such as a graphics processing unit for processing and synthesis, and the reverse tone mapping algorithm cannot be executed on all layers that need to add the high dynamic range effect, resulting in the loss of the high dynamic range effect of the video and problems such as obvious color differences during video playback, affecting the user's viewing experience.
[0065] Based on this, the exemplary embodiments of the present disclosure provide a video processing method. In the case where there is a first initial layer that has not been added with the high dynamic range effect after image processing, by performing the first effect addition process on the synthesized layer, the obtained target layer has the high dynamic range effect, which can avoid the loss of the original high dynamic range effect of the video, ensure the color effect during video playback, and improve the user's viewing experience.
[0066] In an exemplary embodiment, a video processing method is provided. Referring to Figure 1 as shown, the video processing method includes:
[0067] S100. When the electronic device plays a video, perform image processing and merging on multiple initial layers of the frame image in the video source data to obtain a synthesized layer, and the multiple initial layers include a first initial layer that needs to add the high dynamic range effect.
[0068] In step S100, the video played by the electronic device can be, for example, a high-dynamic-range video. The electronic device can play the video through, for example, a video application or an album application. The video source data of the played video includes multiple frame images. Each frame image can be composed of multiple initial layers. When the played video is a high-dynamic-range video, the multiple initial layers include a first initial layer that needs to add a high-dynamic-range effect and a second initial layer that does not need to add a high-dynamic-range effect. Each first initial layer needs to be specially processed to add a high-dynamic-range effect before the video can have a high-dynamic-range effect. The electronic device performs image processing and merging on the multiple initial layers of each frame image to obtain a target layer. The image processing and merging of the initial layers can be performed independently or jointly by the first chip and the second chip of the electronic device.
[0069] S200. If there is a first initial layer that has not added a high-dynamic-range effect after image processing, perform a first effect addition process on the composite layer to obtain a target layer. The first effect addition process is used to add a high-dynamic-range effect to the composite layer.
[0070] In step S200, after the image processing process is completed, the first initial layers included in the initial layers may have all added a high-dynamic-range effect, partially added a high-dynamic-range effect, or not added a high-dynamic-range effect at all. The electronic device can judge whether there is a first initial layer that has not added a high-dynamic-range effect. When there is a first initial layer that has not added a high-dynamic-range effect, if the composite layer is directly used as the finally displayed layer, the original high-dynamic-range effect of the video will be lost. At this time, perform a first effect addition process on the composite layer to add a high-dynamic-range effect to the composite layer to obtain a target layer, ensuring that the subsequent displayed target layer has a high-dynamic-range effect.
[0071] The first effect addition process can be, for example, performing an inverse tone mapping algorithm on the composite layer. The inverse tone mapping algorithm includes various types such as a global algorithm, a classification algorithm, and an extended mapping algorithm, and can convert a low-dynamic-range or standard-dynamic-range image into a high-dynamic-range image, making the composite layer have a high-dynamic-range effect. The first effect addition process can be performed, for example, by the first chip of the electronic device or a third chip independent of the first chip and the second chip.
[0072] S300. Display the target layer.
[0073] In step S300, send the target layer with a high-dynamic-range effect to the display of the electronic device for display to realize the display of the target layer, so that a video with a high-dynamic-range effect can be played through a display panel.
[0074] In this embodiment, a composite layer is obtained by performing image processing and merging on multiple initial layers of the frame images in the video source data. When there is a first initial layer that has not been added with a high-dynamic range effect after the image processing, a first effect addition process is performed on the composite layer, so that the obtained target layer has a high-dynamic range effect, which can avoid the video losing its original high-dynamic range effect, ensure the color effect during video playback, and improve the user's viewing experience.
[0075] In some embodiments, referring to Figure 2 as shown, performing image processing and merging on each initial layer of the frame images in the video source data to obtain a composite layer includes:
[0076] S110. Performing image processing on some of the multiple initial layers respectively through a first chip to obtain at least one first intermediate layer.
[0077] In step S110, the electronic device performs image processing on some of the initial layers respectively through the first chip to obtain first intermediate layers corresponding to the number of initial layers processed by the first chip. Exemplarily, the first chip is a central processing unit. The electronic device performs image processing on some of the initial layers respectively through the Device and HWComposer modules of the central processing unit. When the first initial layer that needs to be added with a high-dynamic range effect is included in these initial layers, a second effect addition process can be performed on the first initial layer through the Device and HWComposer modules to add a high-dynamic range effect to the first initial layer. The second effect addition process can be, for example, an inverse tone mapping algorithm that is the same as or different from the first effect addition process. The first initial layer handed over to the first chip for processing can all complete the addition of the high-dynamic range effect.
[0078] S120. Performing image processing and merging on another part of the multiple initial layers through a second chip to obtain a second intermediate layer.
[0079] In step S120, when the display screen rotates or the screen is reduced or enlarged by a ratio exceeding a certain range, it will cause the processing capacity of the first chip to be insufficient. The electronic device can send another part of the initial layers to the second chip when the processing capacity of the first chip is insufficient, and perform image processing on another part of the initial layers through the second chip and merge the processed layers to obtain a merged second intermediate layer. Exemplarily, the second chip is a graphics processing unit. The electronic device performs image processing on another part of the initial layers respectively and merges them.
[0080] S130. Merging at least one first intermediate layer and the second intermediate layer to obtain a composite layer;
[0081] In step S130, exemplarily, the graphics processor sends the obtained second intermediate layer to the central processing unit, and the central processing unit may merge at least one first intermediate layer and the second intermediate layer through the HWComposer module to obtain a composite layer.
[0082] It can be understood that when the initial layer processed by the second chip includes the first initial layer that needs to add a high dynamic range effect, the second chip cannot perform special effect addition processing on the first initial layer, so that neither the first initial layer after image processing by the second chip nor the merged second intermediate layer has a high dynamic range effect. At this time, if the composite layer obtained by directly merging the second intermediate layer and the first intermediate layer is used as the final display layer, the high dynamic range effect of the video will be lost. Therefore, it is necessary to perform the first effect addition processing on the composite layer according to the above method.
[0083] In this embodiment, the first chip performs image processing on a part of the initial layers respectively to obtain at least one first intermediate layer, and the second chip performs image processing and merging on another part of the initial layers among the multiple initial layers to obtain a second intermediate layer, and then merges at least one first intermediate layer and the second intermediate layer. When the processing capacity of the first chip is insufficient, etc., the second chip shares the processing and synthesis pressure of the initial layers, ensuring the stable output of the composite layer.
[0084] It should be noted that the electronic device can not only hand over some initial layers to the second chip for image processing and merging when the processing capacity of the first chip is insufficient, but also make the two parts of the initial layers perform image processing through the first chip and the second chip respectively in an actively controlled manner. The electronic device can also hand over all the initial layers to the first chip or hand over all the initial layers to the second chip for image processing and merging. When all the initial layers are processed by the first chip, it can ensure that each first initial layer realizes the addition of the high dynamic range effect, and then there is no need to perform the first effect addition processing on the composite layer subsequently. When all the initial layers are processed by the second chip, each first initial layer does not add the high dynamic range effect, and then it is necessary to perform the first effect addition processing on the composite layer subsequently.
[0085] In some embodiments, referring to Figure 3 As shown, before performing image processing and merging on multiple initial layers of the frame image in the video source data to obtain a composite layer, the video processing method further includes:
[0086] S410, set a to-be-processed flag for the first initial layer among the multiple initial layers.
[0087] In step S410, the electronic device sets a to-be-processed flag for the first initial layer among the initial layers of each frame image. The to-be-processed flag is used to indicate that the layer with this to-be-processed flag needs to add a high dynamic range effect. Exemplarily, the electronic device can, for example, obtain the layer information of each initial layer, determine whether the initial layer is the first initial layer that needs to add a high dynamic range effect based on the layer information, and then set the to-be-processed flag for the first initial layer through the SurfaceFlinger service. Among them, the SurfaceFlinger service is in the framework layer (Framework) of the electronic device and is used for the management and control of graphics display. It can collect and manage all the image data drawn by application programs and then display this data on the physical screen.
[0088] S420. After the first chip performs a second effect addition process on the first initial layer, clear the to-be-processed flag of the first initial layer or modify the to-be-processed flag to a processed flag.
[0089] In step S420, after the first chip performs a second effect addition process on the first initial layer, this part of the first initial layer has completed the addition of the high dynamic range effect. The to-be-processed flag of this part of the first initial layer can be cleared, or the to-be-processed flag can be modified to a processed flag to indicate that the first intermediate layer after the initial layer completes image processing no longer needs to add a high dynamic range effect, which is convenient for judging whether there is a first initial layer that has not added a high dynamic range effect after image processing.
[0090] S430. If the initial layer processed by the second chip includes the first initial layer, set a to-be-processed flag for the second intermediate layer.
[0091] In step S430, if the initial layer processed by the second chip includes the first initial layer, since the second chip cannot perform an effect addition process on the first initial layer, it is necessary to set a to-be-processed flag for the synthesized second intermediate layer. That is, when there is at least one first initial layer in the initial layers processed by the second chip, set a to-be-processed flag for the synthesized second intermediate layer to indicate that the first initial layer is included in the initial layers that make up the second intermediate layer, which is convenient for judging whether there is a first initial layer that has not added a high dynamic range effect after image processing.
[0092] In this embodiment, by setting a to-be-processed flag for the first initial layer among multiple initial layers, clearing or modifying the to-be-processed flag after the first chip performs the second effect addition process on the first initial layer, and setting a to-be-processed flag for the second intermediate layer when the initial layer processed by the second chip includes the first initial layer, it is possible to represent, through the to-be-processed flag, the first initial layer for which the high dynamic range effect needs to be added and has not been added yet. This facilitates the electronic device to identify the first intermediate layer and the second intermediate layer, so as to quickly and accurately determine whether there is a first initial layer without the high dynamic range effect added after image processing based on whether the first intermediate layer and the second intermediate layer carry the to-be-processed flag, providing a basis for whether to perform the first effect addition process on the composite layer.
[0093] In some embodiments, merging at least one first intermediate layer and a second intermediate layer to obtain a composite layer includes: if there is a layer with a to-be-processed flag among at least one first intermediate layer and the second intermediate layer, setting a to-be-processed flag for the composite layer.
[0094] When merging at least one first intermediate layer and the second intermediate layer, if there is a layer with a to-be-processed flag among at least one first intermediate layer and the second intermediate layer, it means that the second effect addition process was not performed when the first chip performed image processing on the first initial layer, or the initial layer processed by the second chip includes the first initial layer. If the composite layer directly obtained by merging the first intermediate layer and the second intermediate layer is used as the final layer to be displayed, it will cause the loss of the high dynamic range effect in the video. Then, a to-be-processed flag is set for the composite layer, which is used to represent that the composite layer with the to-be-processed flag needs to perform the first effect addition process. After image processing, by identifying whether the composite layer carries the to-be-processed flag, it can be determined whether there is a first initial layer without the high dynamic range effect added, and then it can be decided whether to perform the first effect addition process on the composite layer.
[0095] In this embodiment, in the case where there is a layer with a to-be-processed flag among at least one first intermediate layer and the second intermediate layer, by setting a to-be-processed flag for the composite layer, it is convenient for the electronic device to identify the composite layer, so as to quickly and accurately determine whether there is a first initial layer without the high dynamic range effect added after image processing based on whether the composite layer carries the to-be-processed flag, providing a basis for whether to perform the first effect addition process on the composite layer.
[0096] In some embodiments, if there is a first initial layer without the high dynamic range effect added after image processing, performing a first effect addition process on the composite layer to obtain a target layer includes: if the composite layer carries a to-be-processed flag, determining that there is a first initial layer without the high dynamic range effect added, and performing a first effect addition process on the composite layer to obtain a target layer.
[0097] The electronic device can identify the composite layer. If it identifies that the composite layer has a to-be-processed flag, it can determine that there is a first initial layer without the high dynamic range effect added after image processing. Then, the electronic device performs the first effect addition process on the composite layer to obtain a target layer with the high dynamic range effect.
[0098] In this embodiment, using the fact that the composite layer has a to-be-processed flag as the basis for determining the existence of the first initial layer without the high dynamic range effect added can quickly and accurately determine whether there is a first initial layer without the high dynamic range effect added after image processing by identifying whether the composite layer has a to-be-processed flag, providing a basis for whether to perform the first effect addition process on the composite layer.
[0099] As can be seen from the above embodiments, the first initial layer in the initial layer is set with a to-be-processed flag, and the to-be-processed flag can be cleared or modified after the first chip performs the second effect addition process on the first initial layer. If the first initial layer is processed by the first chip and the second effect addition process is not performed, the first intermediate layer will have a to-be-processed flag. If the second initial layer is processed by the second chip, the second intermediate layer will have a to-be-processed flag. When at least one of the first intermediate layer and the second intermediate layer has a to-be-processed flag, the composite layer after merging will be set with a to-be-processed flag to instruct the electronic device to perform the first effect addition process on the composite layer. That is, in the case where any first initial layer does not have the high dynamic range effect added, its to-be-processed flag will be retained in the image processing and the merged composite layer, and the first effect addition process will be performed on the composite layer with the to-be-processed flag. This can fully avoid the loss of the high dynamic range effect in the video due to any first initial layer not having the high dynamic range effect added, ensure the color effect during video playback, and improve the user's viewing experience.
[0100] In some embodiments, the first effect addition process performed on the composite layer can be the same as the second effect addition process performed by the first chip on the first initial layer during the image processing process.
[0101] In other embodiments, referring to Figure 4 as shown, performing the first effect addition process on the composite layer to obtain the target layer includes:
[0102] S210. Perform linearization processing on the composite layer to obtain a first layer.
[0103] In step S210, during the process of image processing and merging of the initial layer, there are steps of non-linear operations, such that the data or image signals in the composite layer do not have a linear relationship with the radiance in the real scene. Therefore, when performing the first effect addition process such as the inverse tone mapping algorithm on the composite layer, it is necessary to first perform a linearization operation on the composite layer to convert the data in the composite layer into a linear space, facilitating subsequent processing of the color data therein.
[0104] S220. Preprocess the first layer to obtain a second layer.
[0105] In step S220, preprocess the first layer after linearization processing of the composite layer. The preprocessing can include, for example, processing of color and brightness, enabling the obtained second layer to have good color and brightness effects.
[0106] S230. Perform filtering processing on the second layer to obtain a third layer.
[0107] In step S230, the second layer can be filtered through a filter to eliminate or compress the noise in the second layer while retaining the detail features of the layer, thereby improving the image quality of the second layer.
[0108] S240. Perform tone curve mapping processing on the third layer to obtain a fourth layer.
[0109] In step S240, use an exponential function or a logarithmic function to perform tone curve mapping processing on the third layer to obtain a fourth layer. Exemplarily, the third chip can be used to save the content completed by the tone curve mapping of the previous image frame, and the subsequent image frame can be compensated through, for example, an image color clustering algorithm to prevent overexposure or underexposure problems.
[0110] S250. Perform gamma correction processing on the fourth layer to obtain the target layer.
[0111] In step S250, since the human eye's vision is non-linear and the human eye is more sensitive to changes in the dark areas, and at the same time, the means of storing, transmitting, and displaying images have bandwidth limitations, a larger bandwidth should be given to the dark areas during storage. Through gamma correction, more storage bit widths can be allocated to the dark areas. Therefore, perform gamma correction on the fourth layer to complete the entire first effect addition process and obtain the target layer for final display.
[0112] In this embodiment, the first effect adding processing of the composite layer is completed by performing linearization processing, preprocessing, filtering processing, tone curve mapping processing and gamma correction processing on the composite layer in sequence, and the output of the target layer is realized, so that the target layer sent for display can have a high dynamic range effect, ensuring the color effect during video playback and improving the user's viewing experience.
[0113] In some embodiments, the second layer is filtered by using a unilateral filtering method. For example, a unilateral filter of a fixed size may be repeatedly used to achieve a filtering effect by an iterative method.
[0114] In some other embodiments, filtering the second layer includes: performing bilateral filtering on the second layer to obtain a third layer.
[0115] When filtering the second layer, a bilateral filter is used to perform bilateral filtering on the second layer. Bilateral filtering only requires one filtering calculation and can simultaneously consider spatial information and range information, while removing noise in the layer, retaining the edge information of the image as much as possible, thereby improving the image quality of the third layer.
[0116] In this embodiment, the second layer is filtered by bilateral filtering, which can take into account both noise removal and retention of layer details, and the third layer can be acquired through one filtering calculation, thereby optimizing the number of calculations and reducing performance waste.
[0117] In some embodiments, performing a first effect adding process on the composite layer to obtain a target layer includes: performing a first effect adding process on the composite layer through a third chip to obtain the target layer.
[0118] As mentioned above, the execution of the first effect adding process can be performed by a third chip of the electronic device that is independent of the first chip and the second chip. The third chip can be, for example, an image enhancement chip, and the third chip can execute an inverse tone mapping algorithm to perform the first effect adding process on the composite layer. Exemplarily, after the first chip merges the first intermediate layer and the second intermediate layer into a composite layer through the HWComposer module, the composite layer is sent to the third chip, and the first effect adding process is performed on the composite layer through the third chip to obtain a target layer with a high dynamic range effect. The third chip then sends the target layer to the display for display.
[0119] In this embodiment, the third chip performs the first effect addition process on the composite layer, enabling the first chip and the second chip to complete the image processing of the current image frame after obtaining the composite layer. The subsequent first effect addition process and display sending of the composite layer can be completed by the third chip independent of the first chip and the second chip, releasing the processing pressure of the first chip, ensuring the stable output of the target layer, guaranteeing the color effect during video playback, and enhancing the user's viewing experience.
[0120] In some embodiments, the first chip includes a central processing unit, the second chip includes a graphics processing unit, and the third chip includes an image enhancement chip.
[0121] As mentioned above, the first chip includes a central processing unit, the second chip includes a graphics processing unit, and the third chip includes an image enhancement chip. Refer to Figure 5 shown, the central processing unit can perform the second effect addition process on the first initial layer through its Device and HWComposer modules, and can merge the first intermediate layer and the second intermediate layer. The graphics processing unit can merge at least part of the initial layers when the processing capacity of the central processing unit is limited, and send the merged second intermediate layer back to the central processing unit. The image enhancement chip can perform the first effect addition process on the composite layer in a specific scenario and send the obtained target layer to the display.
[0122] In this embodiment, taking the central processing unit, the graphics processing unit, and the image enhancement chip as the first chip, the second chip, and the third chip respectively, can realize the respective processing tasks of the first chip, the second chip, and the third chip through the central processing unit, the graphics processing unit, and the image enhancement chip, so as to complete the output of the final target layer through the mutual cooperation of the three, ensure that the target layer has a high dynamic range effect, guarantee the color effect during video playback, and enhance the user's viewing experience.
[0123] In an exemplary embodiment, a video processing method is provided. Refer to Figure 6 shown, the video processing method includes:
[0124] S1. When the electronic device plays a video, set a to-be-processed flag for the first initial layer among the multiple initial layers of the frame image in the video source data, where the first initial layer is the initial layer that needs to add a high dynamic range effect;
[0125] S2. Perform image processing on some of the multiple initial layers through the central processing unit to obtain at least one first intermediate layer;
[0126] S3. After the central processing unit performs the second effect addition process on the first initial layer, clear the to-be-processed flag of the first initial layer or modify the to-be-processed flag to a processed flag;
[0127] S4. Use the graphics processing unit to perform image processing and merging on another part of the initial layers among the multiple initial layers to obtain a second intermediate layer;
[0128] S5. If the initial layers processed by the graphics processing unit include the first initial layer, set a to-be-processed flag for the second intermediate layer;
[0129] S6. Use the central processing unit to merge at least one first intermediate layer and the second intermediate layer to obtain a composite layer;
[0130] S7. If there is a layer with a to-be-processed flag among at least one first intermediate layer and the second intermediate layer, set a to-be-processed flag for the composite layer;
[0131] S8. If the composite layer has a to-be-processed flag, determine that there is a first initial layer to which the high dynamic range effect has not been added;
[0132] S9. Use the image enhancement chip to perform linearization processing on the composite layer to obtain a first layer;
[0133] S10. Perform preprocessing on the first layer to obtain a second layer;
[0134] S11. Perform bilateral filtering processing on the second layer to obtain a third layer;
[0135] S12. Perform tone curve mapping processing on the third layer to obtain a fourth layer;
[0136] S13. Perform gamma correction processing on the fourth layer to obtain the target layer;
[0137] S14. Display the target layer.
[0138] In this embodiment, by performing image processing and merging on multiple initial layers of the frame images in the video source data to obtain a composite layer, and in the case that there is a first initial layer to which the high dynamic range effect has not been added after image processing, by performing the first effect addition process on the composite layer, the obtained target layer has a high dynamic range effect, which can avoid the video losing the original high dynamic range effect, ensure the color effect during video playback, and improve the user's viewing experience.
[0139] In an exemplary embodiment, a video processing device is provided, which is applied to an electronic device. Refer to Figure 7As shown in the figure, the video processing device includes a synthesis module 10, an effect adding module 20, and a display module 30. The synthesis module 10 is configured to perform image processing and merging on multiple initial layers of frame images in video source data when the electronic device plays a video, so as to obtain a synthesized layer. The multiple initial layers include a first initial layer that needs to add a high dynamic range effect. The effect adding module 20 is configured to, if there is a first initial layer that has not added a high dynamic range effect after image processing, perform a first effect adding process on the synthesized layer to obtain a target layer. The first effect adding process is used to add a high dynamic range effect to the synthesized layer. The display module 30 is configured to display the target layer.
[0140] In this embodiment, the synthesis module 10 performs image processing and merging on multiple initial layers of frame images in the video source data to obtain a synthesized layer. When there is a first initial layer that has not added a high dynamic range effect after image processing, the effect adding module 20 performs a first effect adding process on the synthesized layer, so that the obtained target layer has a high dynamic range effect. Then, the display module 30 displays the target layer, which can avoid the video losing the original high dynamic range effect, ensure the color effect during the video playback process, and improve the user's viewing experience.
[0141] In one embodiment, the synthesis module 10 is further configured to: perform image processing on some of the multiple initial layers through a first chip respectively to obtain at least one first intermediate layer. Perform image processing and merging on another part of the multiple initial layers through a second chip to obtain a second intermediate layer. Merge the at least one first intermediate layer and the second intermediate layer to obtain a synthesized layer. When the first initial layer is included in the initial layers processed by the first chip, the first chip performs a second effect adding process on the first initial layer. The second effect adding process is used to add a high dynamic range effect to the first initial layer.
[0142] In some embodiments, the video processing device further includes an identification module. The identification module is configured to: set a to-be-processed identification for the first initial layer among the multiple initial layers. After the first chip performs a second effect adding process on the first initial layer, clear the to-be-processed identification of the first initial layer or modify the to-be-processed identification to a processed identification. If the first initial layer is included in the initial layers processed by the second chip, set a to-be-processed identification for the second intermediate layer.
[0143] In one embodiment, the identification module is further configured to: if there is a layer with a to-be-processed identification among the at least one first intermediate layer and the second intermediate layer, set a to-be-processed identification for the synthesized layer.
[0144] In one embodiment, the effect adding module 20 is further configured to: if the composite layer has a to-be-processed identifier, determine a first initial layer without a high dynamic range effect added thereto, and perform a first effect adding process on the composite layer to obtain a target layer.
[0145] In one embodiment, the effect adding module 20 is further configured to: perform a linearization process on the composite layer to obtain a first layer. Perform a preprocessing on the first layer to obtain a second layer. Perform a filtering process on the second layer to obtain a third layer. Perform a tone curve mapping process on the third layer to obtain a fourth layer. Perform a gamma correction process on the fourth layer to obtain a target layer.
[0146] In one embodiment, the effect adding module 20 is further configured to: perform a bilateral filtering process on the second layer to obtain a third layer.
[0147] In one embodiment, the effect adding module 20 is further configured to: perform a first effect adding process on the composite layer through a third chip to obtain a target layer.
[0148] In one embodiment, the first chip includes a central processing unit, the second chip includes a graphics processing unit, and the third chip includes an image enhancement chip.
[0149] In an exemplary embodiment, an electronic device is provided. The electronic device may be, for example, a mobile phone, a tablet computer, or the like.
[0150] Reference Figure 8 As shown, the electronic device may include one or more of the following components: a processing component 101, a memory 102, a power component 103, a multimedia component 104, an audio component 105, an input / output (I / O) interface 106, a sensor component 107, and a communication component 108.
[0151] The processing component 101 generally controls the overall operation of the electronic device, such as operations associated with display, telephone call, data communication, camera operation, and recording operation. The processing component 101 may include one or more processors 109 to execute instructions to complete all or part of the steps of the above methods. In addition, the processing component 101 may include one or more modules to facilitate the interaction between the processing component 101 and other components. For example, the processing component 101 may include a multimedia module to facilitate the interaction between the multimedia component 104 and the processing component 101.
[0152] The memory 102 is configured to store various types of data to support the operation of the electronic device. Examples of such data include instructions for any application or method operating on the electronic device, contact data, phone book data, messages, pictures, audio, and the like. The memory 102 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, a magnetic disk, or an optical disk.
[0153] The power component 103 provides power to various components of the electronic device. The power component 103 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power for the electronic device.
[0154] The multimedia component 104 includes a screen that provides an output interface between the electronic device and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can sense not only the boundaries of the touch or swipe actions but also detect the duration and pressure associated with the touch or swipe operations. In some embodiments, the multimedia component 104 includes a front camera and / or a rear camera. When the electronic device is in an operating mode, such as a shooting mode or an audio mode, the front camera and / or the rear camera can receive external multimedia data. Each of the front camera and the rear camera can be a fixed optical lens system or have focal length and optical zoom capabilities.
[0155] The audio component 105 is configured to output and / or input audio signals. For example, the audio component 105 includes a microphone (MIC) that is configured to receive external audio signals when the electronic device is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals can be further stored in the memory 102 or transmitted via the communication component 108. In some embodiments, the audio component 105 further includes a speaker for outputting audio signals.
[0156] The I / O interface 106 provides an interface between the processing component 101 and a peripheral interface module, which can be a keyboard, a click wheel, buttons, etc. These buttons can include, but are not limited to: a home button, a volume button, a power button, and a lock button.
[0157] The sensor assembly 107 includes one or more sensors for providing a status assessment of various aspects of the electronic device. For example, the sensor assembly 107 can detect the on / off state of the electronic device, the relative positioning of components, such as the display and keypad of the electronic device, the sensor assembly 107 can also detect a change in the position of the electronic device or a component of the electronic device, the presence or absence of user contact with the electronic device, the orientation or acceleration / deceleration of the electronic device, and the temperature change of the electronic device. The sensor assembly 107 can include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 107 can also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 107 can further include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.
[0158] The communication component 108 is configured to facilitate communication between the electronic device and other devices in a wired or wireless manner. The device can access a wireless network based on communication standards, such as WiFi, 2G, or 3G, or a combination thereof. In an exemplary embodiment, the communication component 108 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 108 further includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.
[0159] In an exemplary embodiment, the electronic device can be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components for performing the video processing method applied to the electronic device as described above.
[0160] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as the memory 102 including instructions, and the above instructions can be executed by the processor 109 of the electronic device to complete the video processing method applied to the electronic device as described above. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device, etc. When the instructions in the storage medium are executed by the processor 109 of the electronic device, the electronic device is enabled to execute the video processing method shown in the above embodiments.
[0161] Other embodiments of the present invention will be readily apparent to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention, which follow the general principles of the invention and include known common general knowledge or conventional technical means in the technical field not disclosed in this disclosure. The specification and examples are only to be regarded as exemplary, and the true scope and spirit of the invention are pointed out by the following claims.
[0162] It should be understood that the present invention is not limited to the exact structures already described and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present invention is only limited by the appended claims.
Claims
1. A video processing method, applied to an electronic device, characterized in that, The video processing method includes: When the electronic device plays a video, performing image processing and merging on multiple initial layers of a frame image in video source data to obtain a composite layer, where the multiple initial layers include a first initial layer that needs to add a high dynamic range effect; If there is the first initial layer that has not added the high dynamic range effect after the image processing, performing a first effect addition process on the composite layer to obtain a target layer, where the first effect addition process is used to add the high dynamic range effect to the composite layer; Displaying the target layer.
2. The video processing method according to claim 1, wherein The performing image processing and merging on each initial layer of the frame image in the video source data to obtain a composite layer includes: Performing image processing on some of the initial layers in the multiple initial layers through a first chip to obtain at least one first intermediate layer; Performing image processing and merging on another part of the initial layers in the multiple initial layers through a second chip to obtain a second intermediate layer; Merging the at least one first intermediate layer and the second intermediate layer to obtain the composite layer; Wherein, when the first initial layer is included in the initial layers processed by the first chip, the first chip performs a second effect addition process on the first initial layer, and the second effect addition process is used to add the high dynamic range effect to the first initial layer.
3. The video processing method according to claim 2, wherein Before performing image processing and merging on multiple initial layers of a frame image in the video source data to obtain a composite layer, the video processing method further includes: Setting a to-be-processed flag for the first initial layer in the multiple initial layers; After the first chip performs the second effect addition process on the first initial layer, clearing the to-be-processed flag of the first initial layer or modifying the to-be-processed flag to a processed flag; If the first initial layer is included in the initial layers processed by the second chip, setting the to-be-processed flag for the second intermediate layer.
4. The video processing method according to claim 3, wherein The merging the at least one first intermediate layer and the second intermediate layer to obtain the composite layer includes: If there is a layer with the to-be-processed flag among the at least one first intermediate layer and the second intermediate layer, setting the to-be-processed flag for the composite layer.
5. The video processing method according to claim 4, wherein The if there is the first initial layer that has not added the high dynamic range effect after the image processing, performing a first effect addition process on the composite layer to obtain a target layer includes: If the composite layer has the to-be-processed flag, determining that there is the first initial layer that has not added the high dynamic range effect, and performing a first effect addition process on the composite layer to obtain a target layer.
6. The video processing method according to any one of claims 1-5, characterized in that, The performing a first effect addition process on the composite layer to obtain a target layer includes: Performing linearization processing on the composite layer to obtain a first layer; Performing preprocessing on the first layer to obtain a second layer; Performing filtering processing on the second layer to obtain a third layer; Performing tone curve mapping processing on the third layer to obtain a fourth layer; Perform gamma correction processing on the fourth layer to obtain the target layer.
7. The video processing method according to claim 6, wherein The filtering processing on the second layer includes: Perform bilateral filtering processing on the second layer to obtain the third layer.
8. The video processing method according to any one of claims 2-5, characterized in that, The first effect addition processing performed on the composite layer to obtain the target layer includes: Perform the first effect addition processing on the composite layer through the third chip to obtain the target layer.
9. The video processing method according to claim 8, wherein The first chip includes a central processing unit, the second chip includes a graphics processing unit, and the third chip includes an image enhancement chip.
10. A video processing device, applied to an electronic device, characterized in that, The video processing device includes: A synthesis module, which is used to perform image processing and merging on multiple initial layers of frame images in video source data when the electronic device plays a video, to obtain a composite layer, and the multiple initial layers include a first initial layer that needs to add a high dynamic range effect; An effect addition module, which is used to perform a first effect addition processing on the composite layer to obtain a target layer if there is a first initial layer that has not added the high dynamic range effect after the image processing, and the first effect addition processing is used to add the high dynamic range effect to the composite layer; A display module, which is used to display the target layer.
11. An electronic device, characterized in that, The electronic device includes: A processor; A memory for storing instructions executable by the processor; Wherein, the processor is configured to: When the electronic device plays a video, perform image processing and merging on multiple initial layers of frame images in video source data to obtain a composite layer, and the multiple initial layers include a first initial layer that needs to add a high dynamic range effect; If there is a first initial layer that has not added the high dynamic range effect after the image processing, perform a first effect addition processing on the composite layer to obtain a target layer, and the first effect addition processing is used to add the high dynamic range effect to the composite layer; Display the target layer.
12. A non-transitory computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device can execute a video processing method, and the video processing method includes: When the electronic device plays a video, perform image processing and merging on multiple initial layers of frame images in video source data to obtain a composite layer, and the multiple initial layers include a first initial layer that needs to add a high dynamic range effect; If there is a first initial layer that has not added the high dynamic range effect after the image processing, perform a first effect addition processing on the composite layer to obtain a target layer, and the first effect addition processing is used to add the high dynamic range effect to the composite layer; Display the target layer.