Video processing method, device, equipment and storage medium

Abstract

The present application discloses a video processing method, apparatus, device and storage medium, which belongs to the field of image processing technology. The method includes: obtaining multiple frames of video images in a video. For any one frame of video images in the multiple frames of video images, when there is a preset target in this frame of video image, the color temperature of this frame of video image is obtained. Afterwards, if the color temperature of this frame of video is the first color temperature, the global LUT and the first correction LUT are used to process this frame of video image. If the color temperature of this frame of video is the second color temperature, the global LUT and the second correction LUT are used to process this frame of video image. In this way, when the color temperature of the video image is different, different correction LUTs can be used to perform color correction on the area where the preset target is located in the video image, so that the video image processed by the global LUT and the correction LUT can not only have the filter effect corresponding to the global LUT, but also effectively avoid the color cast problem of the preset target.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a video processing method, apparatus, device and storage medium. Background Art

[0002] With the development of terminal technology, terminals have gradually integrated communication, photography, and audio and video functions, becoming an indispensable part of people's daily lives. Users can use terminals to shoot videos and record every detail of their lives.

[0003] Currently, the terminal supports video shooting in Movie mode, which provides a color lookup table (LUT) for achieving a cinematic look. When users shoot video in Movie mode, the terminal can use the LUT to process the captured video image, resulting in a more professional, cinematic effect.

[0004] In the above method, LUT is effective for the entire video image. However, due to the influence of light in the shooting environment, the video image processed with LUT may have color cast problems. Summary of the Invention

[0005] This application provides a video processing method, apparatus, device, and storage medium that can effectively prevent color cast in video images. The technical solution is as follows:

[0006] In a first aspect, a video processing method is provided. In this method, multiple frames of video images are obtained from a video. For any frame of the multiple frames, the color temperature of the frame can be obtained if a preset target exists in the frame. If the color temperature of the frame is a first color temperature, the frame is processed using a global LUT and a first correction LUT. If the color temperature of the frame is a second color temperature, the frame is processed using the global LUT and a second correction LUT.

[0007] Among them, the global LUT is used to process the video image as a whole, the first correction LUT and the second correction LUT are both used to perform color correction on the area where the preset target is located in the video image, the first color temperature is different from the second color temperature, and the first correction LUT is different from the second correction LUT.

[0008] The preset target can be set in advance. Optionally, the preset target in this application can be a target with a memory color, such as a portrait, where the memory color of the portrait is skin color. Of course, the preset target can also be other targets with memory colors, such as the sky, water, green plants, etc.

[0009] Optionally, before processing the video, the terminal can predetermine a global LUT for processing the entire video image. Optionally, this global LUT can be manually selected by the user, or automatically selected by the terminal based on the image content displayed in the camera preview interface. This global LUT is a LUT used to process the entire video frame.

[0010] For example, if the terminal detects a selection operation for movie mode, it displays the camera preview interface in movie mode. The camera preview interface includes a LUT control, which is used to activate the LUT function. Subsequently, if the terminal detects a selection operation for the LUT control, it displays a LUT bar on the camera preview interface. The LUT bar includes multiple LUT icons. These multiple LUT icons are icons of each global LUT in multiple pre-set global LUTs. Subsequently, if the terminal detects a selection operation for one of the multiple LUT icons, it obtains the LUT corresponding to this LUT icon as the global LUT.

[0011] The first correction LUT is a LUT used to process the area containing the preset target in the current frame of video. The first correction LUT is used to correct the color of the preset target under illumination with a light source having a first color temperature. Thus, the first correction LUT can perform color correction on the area containing the preset target in the current frame of video. In this case, using the first correction LUT to process the area containing the preset target in the current frame of video can improve the color cast in the area containing the preset target in the current frame of video.

[0012] The second correction LUT is used to process the area containing the preset target in the current frame of video. The second correction LUT is used to correct the color of the preset target under the illumination of a light source having a second color temperature. Thus, the second correction LUT can perform color correction on the area containing the preset target in the current frame of video. In this case, using the second correction LUT to process the area containing the preset target in the current frame of video can improve the color cast in the area containing the preset target.

[0013] In this application, when the color temperature of the video image is different, different correction LUTs can be used to perform color correction on the area where the preset target is located in the video image. In this way, the flexibility and accuracy of color correction can be improved. The video image processed by the global LUT and the correction LUT can not only have the filter effect corresponding to the global LUT, but also effectively avoid the color cast problem of the preset target.

[0014] As an example, during the shooting process of the camera, multiple frames of video images in the video being shot by the camera can be obtained.

[0015] In this case, the video being captured by the camera is processed in real time. Specifically, each time a frame of video image is obtained from the video being captured by the camera, the global LUT and the correction LUT can be used to process this frame of video image. In this way, when the camera finishes shooting, the processing of multiple frames of video images in the video captured by the camera can be completed to obtain the processed video, which in this application is a video with the filter effect corresponding to the LUT.

[0016] As another example, after the camera finishes shooting, each frame of video image in the video shot by the camera can be obtained.

[0017] In this case, the video captured by the camera is post-processed. Specifically, for each frame of video image obtained from the video captured by the camera, the global LUT and the correction LUT can be used to process this frame of video image. In this way, after completing the processing of multiple frames of video images in the video, the processed video can be obtained. In this application, the video with the filter effect corresponding to the LUT can be obtained.

[0018] The light source irradiating the preset target can be determined based on the color temperature of this frame of video image, and thus a correction LUT capable of correcting the color of the preset target illuminated by the light source can be obtained.

[0019] Optionally, in this application, a software algorithm can be used to obtain the color temperature of this frame of video image. Specifically, the AWB algorithm can be used to estimate the color temperature of the entire frame of video image to obtain the color temperature of this frame of video image; or the AWB algorithm can be used to estimate the color temperature of the area where a preset target is located in this frame of video image to obtain the color temperature of this frame of video image; or the AWB algorithm can be used to estimate the color temperature of neutral gray pixels in this frame of video image to obtain the color temperature of this frame of video image.

[0020] Optionally, in this application, a hardware device may be used to obtain the color temperature of the frame of video image. Specifically, the color temperature of each of the m×n regions in the frame of video image may be detected by a spectral detection sensor, where the spectral detection sensor includes m×n detection elements, where m and n are both positive integers; and the color temperature of the frame of video image may be determined based on the color temperature of each of the m×n regions in the frame of video image.

[0021] As an example, before using the global LUT and the first correction LUT to process this frame of video image, the correction LUT corresponding to the first color temperature can also be obtained from the correspondence between color temperature and correction LUT. The correction LUT corresponding to the first color temperature is the first correction LUT.

[0022] A pre-stored correspondence between color temperatures and correction LUTs can be used to store multiple color temperatures and multiple correction LUTs. Each corresponding correction LUT can be used to correct the color of a preset target under illumination with a light source of that color temperature. This correspondence can be set by technical personnel based on actual needs and experience, such as the user's visual experience requirements.

[0023] It should be noted that if the correction LUT corresponding to the first color temperature is not obtained from the correspondence between color temperature and correction LUT, that is, the correction LUT corresponding to the first color temperature does not exist in the correspondence, then two color temperatures can be determined from the correspondence, and the first color temperature is between these two color temperatures. Then, the correction LUT corresponding to each of the two color temperatures can be obtained from the correspondence, and then the two obtained correction LUTs are interpolated (including but not limited to linear interpolation, etc.) to obtain the first correction LUT.

[0024] As another example, before using the global LUT and the first correction LUT to process this frame of video image, the correction LUT used when processing each frame of k frames of video images that are located before this frame of video image in the multiple frames of video images can also be obtained to obtain k correction LUTs; from the correspondence between color temperature and correction LUT, the correction LUT corresponding to the first color temperature is obtained; and interpolation operation is performed on the k correction LUTs and the correction LUT corresponding to the first color temperature to obtain the first correction LUT.

[0025] k is a positive integer and can be set in advance, for example, k can be 1, 2, 3, etc.

[0026] Since each video frame is processed, and the correction LUT for the processed video image is obtained during this processing, when a video frame is acquired, the video images preceding it have already been processed, and the correction LUTs for these preceding video images are now available. The correction LUTs for these preceding video images can then be interpolated (including but not limited to linear interpolation) with the correction LUT corresponding to the first color temperature to obtain the first correction LUT. This ensures the smoothness of the filter effect after processing the area containing the preset target in the video frame using the first correction LUT, thereby ensuring the temporal smoothness of the color correction effect for the preset target.

[0027] As an example, the global LUT and the first correction LUT are used to process this frame of video image to obtain the processed video image. The operations can be: using the global LUT to process the entire frame of video image to obtain a first image; using the first correction LUT to process the area where the preset target is located in this frame of video image to obtain a second image; and performing image fusion on the first image and the second image to obtain the processed video image.

[0028] This method processes a single frame of video in parallel. In this case, a dual video container can be used to store two video images, both representing the same frame. The global LUT is then used to process one video image as a whole, while the correction LUT is used to process the preset target area in the other video image. The processed results of the two video images are then fused to produce the final processed video image.

[0029] Optionally, before acquiring multiple frames of video images in the video, global LUT weights and correction LUT weights may also be acquired. In this case, the first image and the second image may be fused based on the global LUT weights and the correction LUT weights to obtain a processed video image.

[0030] The global LUT weight is used to indicate the weight of the pixel values ​​of the pixels processed by the global LUT when performing image fusion. That is, the global LUT weight is used to indicate the weight of the pixel values ​​of the pixels in the area where the preset target is located in the first image when performing image fusion. The correction LUT weight is used to indicate the weight of the pixel values ​​of the pixels processed by the correction LUT when performing image fusion. That is, the correction LUT weight is used to indicate the weight of the pixel values ​​of the pixels in the area where the preset target is located in the second image when performing image fusion. The sum of the global LUT weight and the correction LUT weight is 1.

[0031] The operations for obtaining the global LUT weight and the correction LUT weight may include: displaying a slider bar indicating the correction LUT weight; if a sliding operation of a slider in the slider bar is detected, determining the weight corresponding to the position of the slider in the slider bar as the correction LUT weight; and subtracting the correction LUT weight from 1 to obtain the global LUT weight. Alternatively, from the correspondence between the global LUT and the global LUT weight, obtaining the global LUT weight corresponding to the global LUT, and subtracting the global LUT weight from 1 to obtain the correction LUT weight.

[0032] This slider is a control that allows you to change the position of the slider and thus change the weight of the correction LUT. In this case, you can manually set the weight of the correction LUT.

[0033] Because some global LUTs have strong color styles, to ensure the overall color harmony of the video image, the pixel values ​​processed by the global LUT need to have a certain degree of weight when performing image fusion. In this case, the global LUT weight and the correction LUT weight can be automatically determined. In other words, the global LUT weight corresponding to the global LUT can be directly obtained from the corresponding relationship between the global LUT and the global LUT weight, and the correction LUT weight can be calculated accordingly.

[0034] As another example, the global LUT and the first correction LUT are used to process this frame of video image to obtain the processed video image. The operation can be: using the global LUT to process the entire frame of video image to obtain the first image; using the first correction LUT to process the area where the preset target is located in the first image to obtain the processed video image.

[0035] This method processes the video frame in series. In this case, the global LUT is first used to process the entire video frame, and then the correction LUT is used to process the preset target area in the processed image to obtain the processed video image.

[0036] Optionally, the operation of using the first correction LUT to process the area where the preset target is located in the first image can be: segmenting the area where the preset target is located from the first image to obtain a preset target image, using the first correction LUT to process all pixel points in the preset target image to obtain a third image, and performing image fusion on the first image and the third image to obtain a processed video image.

[0037] In a second aspect, a video processing device is provided, wherein the video processing device has the function of implementing the video processing method of the first aspect. The video processing device includes at least one module, wherein the at least one module is used to implement the video processing method of the first aspect.

[0038] In a third aspect, a video processing device is provided. The video processing device includes a processor and a memory, wherein the memory is used to store a program that supports the video processing device in executing the video processing method provided in the first aspect, and to store data involved in implementing the video processing method described in the first aspect. The processor is configured to execute the program stored in the memory. The video processing device may also include a communication bus that establishes a connection between the processor and the memory.

[0039] In a fourth aspect, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium. When the computer-readable storage medium is run on a computer, the computer executes the video processing method described in the first aspect.

[0040] In a fifth aspect, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute the video processing method described in the first aspect.

[0041] The technical effects obtained by the above-mentioned second, third, fourth and fifth aspects are similar to the technical effects obtained by the corresponding technical means in the above-mentioned first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a schematic diagram of the structure of a terminal provided in an embodiment of the present application;

[0043] Figure 2 This is a block diagram of a software system of a terminal provided in an embodiment of the present application;

[0044] Figure 3 is a schematic diagram of a video shooting process in a movie mode provided in an embodiment of the present application;

[0045] Figure 4 This is a schematic diagram of a preview interface before recording in movie mode provided by an embodiment of the present application;

[0046] Figure 5 This is a flow chart of a video processing method provided by an embodiment of the present application;

[0047] Figure 6 is a schematic diagram of a video image and its mask provided in an embodiment of the present application;

[0048] Figure 7 is a schematic diagram of a detection area of ​​a spectral detection sensor in a video image provided by an embodiment of the present application;

[0049] Figure 8 This is a schematic diagram of another preview interface before recording in movie mode provided by an embodiment of the present application;

[0050] Figure 9 is a schematic diagram of a parallel processing process of video images provided in an embodiment of the present application;

[0051] Figure 10 is a schematic diagram of a serial processing process of a video image provided by an embodiment of the present application;

[0052] Figure 11 is a schematic diagram of a video processing process provided by an embodiment of the present application;

[0053] Figure 12 is a flowchart of another video processing method provided by an embodiment of the present application;

[0054] Figure 13 It is a structural diagram of a video processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0056] It should be understood that the “multiple” mentioned in this application refers to two or more. In the description of this application, unless otherwise specified, “ / ” means or, for example, A / B can mean A or B; “and / or” in this article is merely a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in order to facilitate the clear description of the technical solution of this application, words such as “first” and “second” are used to distinguish between identical or similar items with basically the same functions and effects. Those skilled in the art can understand that words such as “first” and “second” do not limit the quantity and execution order, and words such as “first” and “second” do not necessarily limit them to be different.

[0057] The phrases "one embodiment" or "some embodiments" described in this application mean that the specific features, structures, or characteristics described in that embodiment are included in one or more embodiments of the application. Thus, the phrases "in one embodiment," "in some embodiments," "in other embodiments," and "in other embodiments" that appear in different places in this application do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. In addition, the terms "including," "comprising," "having," and their variations all mean "including but not limited to," unless otherwise specifically emphasized.

[0058] The computer device involved in the embodiments of the present application is described below.

[0059] Figure 1 This is a schematic diagram of the structure of a terminal provided by an embodiment of the present application. Figure 1The terminal 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display 194, and a subscriber identity module (SIM) card interface 195, etc. Among them, the sensor module 180 can include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a spectrum detection sensor 180M, etc.

[0060] It should be understood that the structures illustrated in the embodiments of the present application do not constitute a specific limitation on the terminal 100. In other embodiments of the present application, the terminal 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0061] The processor 110 may include one or more processing units, for example, an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0062] The controller may be the nerve center and command center of the terminal 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0063] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0064] The charging management module 140 is configured to receive charging input from a charger. The charger can be either a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 can receive charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 can receive wireless charging input via the wireless charging coil of the terminal 100. While charging the battery 142, the charging management module 140 can also provide power to the terminal 100 via the power management module 141.

[0065] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and provides power to the processor 110, the internal memory 121, the external memory, the display 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage, impedance). In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0066] The wireless communication function of the terminal 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0067] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied on the terminal 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0068] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc. applied on the terminal 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, frequency modulate it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0069] Terminal 100 implements display functions through a GPU, display screen 194, and an application processor. The GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0070] The terminal 100 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor, etc. The terminal 100 may include 1 or N cameras 193, where N is an integer greater than or equal to 2.

[0071] In the embodiment of the present application, the terminal 100 can record video through one or N cameras 193. Specifically, the camera 193 is used to collect video streams. After the camera 193 collects the video streams, it can be transmitted to the ISP for processing.

[0072] As an example, the video image format of the video stream captured by camera 193 is in RAW format. The ISP can convert the RAW format video image in the video stream into a YUV format video image, and then perform basic processing on the YUV format video image, such as adjusting contrast and removing noise. The ISP can then transmit the processed video stream to the application processor.

[0073] The application processor can perform LUT processing on the video image in the received video stream. Specifically, the application processor can perform target detection on the video image. When a preset target is detected in the video image, the application processor obtains the color temperature of the video image and then processes the video image using the global LUT and the correction LUT corresponding to the color temperature of the video image. The correction LUT is used to perform color correction on the area where the preset target is located in the video image. In this way, the video image processed by the global LUT and the correction LUT can not only have the filter effect corresponding to the global LUT, but also effectively avoid the color cast problem of the preset target.

[0074] As an example, the application processor can obtain the color temperature of a video image using the spectral detection sensor 180M. The spectral detection sensor 180M includes m×n detection elements. Thus, the spectral detection sensor 180M can divide the field of view into m×n regions and detect the color temperature of each of these m×n regions. The application processor can obtain the color temperature of each of the m×n regions in the video image detected by the spectral detection sensor 180M and then determine the color temperature of the video image based on the color temperature of each of the m×n regions.

[0075] Furthermore, after processing the video image, the application processor can also display the processed video image on the recording interface through the video codec, GPU and display screen 194 to achieve video preview. In addition, the processed video image can also be stored for playback after the video shooting is completed.

[0076] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the terminal 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement data storage functions. For example, files such as music and videos can be stored on the external memory card.

[0077] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the terminal 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area can store data created by the terminal 100 during use (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0078] The terminal 100 can implement audio functions such as music playback and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D and the application processor.

[0079] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be provided in the processor 110, or some functional modules of the audio module 170 can be provided in the processor 110.

[0080] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to or removed from the terminal 100 by inserting it into or removing it from the SIM card interface 195. The terminal 100 can support 1 or N SIM card interfaces, where N is an integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The terminal 100 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the terminal 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the terminal 100 and cannot be separated from the terminal 100.

[0081] Next, the software system of the terminal 100 will be described.

[0082] The software system of the terminal 100 may adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. In the embodiment of the present application, the Android system with a layered architecture is used as an example to exemplify the software system of the terminal 100.

[0083] Figure 2 This is a block diagram of a software system of a terminal 100 provided in an embodiment of the present application. Figure 2 The layered architecture divides software into several layers, each with distinct roles and responsibilities. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided from top to bottom into the application layer (APP), the application framework layer (FWK), the Android runtime layer, the system layer, and the kernel layer.

[0084] The application layer can include a series of application packages. Figure 2 As shown, the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, etc.

[0085] The application framework layer provides application programming interface (API) and programming framework for the applications in the application layer. The application framework layer includes some predefined functions. Figure 2As shown, the application framework layer may include a window manager, a content provider, a view system, a telephony manager, a resource manager, a notification manager, and the like. The window manager is used to manage window programs. It can obtain the display screen size, determine whether there is a status bar, lock the screen, take screenshots, and so on. The content provider is used to store and retrieve data and make this data accessible to applications. This data may include video, images, audio, dialed and received calls, browsing history and bookmarks, and a phone book. The view system includes visual controls, such as controls for displaying text and controls for displaying images. The view system can be used to construct the application's display interface. The display interface may consist of one or more views, such as a view that displays a text message notification icon, a view that displays text, and a view that displays images. The telephony manager is used to provide communication functions for the terminal 100, such as managing call status (including connected and disconnected calls). The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, and video files. The Notification Manager allows applications to display notifications in the status bar. These messages can be displayed briefly and then disappear automatically, without requiring user interaction. For example, the Notification Manager is used to announce the completion of downloads and message reminders. A Notification Manager can also display notifications in the top status bar of the system as icons or scrolling text, such as notifications from background applications. A Notification Manager can also display notifications in the form of dialog windows on the screen, such as text messages in the status bar, beeps, vibrates electronic devices, or flashes indicator lights.

[0086] The Android Runtime consists of core libraries and a virtual machine (VM). The Android runtime is responsible for scheduling and management of the Android system. The core library consists of two parts: one for Java-based functions and the other for the Android core library. The application layer and application framework layer run in the VM. The VM executes Java files from the application layer and application framework layer as binary files. The VM is responsible for performing functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0087] The system layer can include multiple functional modules, such as: surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc. The surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications. The media library supports playback and recording of a variety of common audio and video formats, as well as static image files. The media library can support a variety of audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc. The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis, and layer processing. The 2D graphics engine is a drawing engine for 2D drawing.

[0088] The kernel layer is the layer between hardware and software. The kernel layer includes at least display driver, camera driver, audio driver, sensor driver, etc.

[0089] Before explaining the embodiments of the present application in detail, the terms involved in the embodiments of the present application are explained.

[0090] 1. User experience (UX): Also known as UX characteristics, in this embodiment of the application, it refers to the user's experience during the shooting process.

[0091] 2. User Interface (UI): This is the interface between an application or operating system and the user for interaction and information exchange, used to convert information between its internal form and a user-interpretable form. A common UI is the graphical user interface (GUI), which uses graphics to display computer operations. For example, a GUI can include various visual interface elements, such as text, icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, and / or navigation bars.

[0092] 3. Movie mode: This is a mode for shooting videos, specifically a special mode within the video recording mode. In the embodiments of the present application, the movie mode includes a LUT function. The LUT function means that when shooting videos in movie mode, the video is given a movie-like texture by using a LUT.

[0093] 4. LUT: A color conversion template, such as a red, green, blue (RGB) mapping table. A LUT can be used to transform the grayscale values ​​of actual pixels (such as thresholding, inversion, contrast adjustment, and linear transformation) to obtain a corresponding grayscale value. This can highlight useful information in the image and enhance the image's light contrast. In other words, using a LUT to process an image can produce an image effect of a specific style, effectively transforming the image into a corresponding filter effect.

[0094] As you can understand, an image consists of many pixels, each represented by an RGB value. A display screen can display the image based on the RGB values ​​of each pixel in the image. In other words, these RGB values ​​dictate how the display screen illuminates, mixing various colors for the user.

[0095] For example, the LUT can be an RGB mapping table for characterizing the correspondence between the RGB values ​​before and after adjustment. For example, the LUT can be the RGB mapping table shown in Table 1 below. When the original RGB value is (14, 22, 24), after the mapping of the LUT shown in Table 1, the output RGB value is (6, 9, 4). When the original RGB value is (61, 34, 67), after the mapping of the LUT shown in Table 1, the output RGB value is (66, 17, 47). When the original RGB value is (94, 14, 171), after the mapping of the LUT shown in Table 1, the output RGB value is (117, 82, 187). When the original RGB value is (241, 216, 222), after the mapping of the LUT shown in Table 1, the output RGB value is (255, 247, 243).

[0096] Table 1

[0097]

[0098] The embodiment of the present application only uses Table 1 above as an example to illustrate the LUT, and Table 1 above does not limit the embodiment of the present application.

[0099] 5. Image segmentation: This is a crucial task in computer vision. Its purpose is to classify each pixel in an image, a pixel-level classification task. For example, conventional segmentation involves separating pixel regions belonging to different objects, a task known as object detection. Semantic segmentation builds on conventional segmentation by identifying the semantic meaning of each pixel region in an image, specifically determining the category of the object to which each pixel region belongs. In other words, semantic segmentation allows for the automatic segmentation and identification of image content, such as identifying people, plants, and buildings.

[0100] The application scenarios involved in the embodiments of the present application are described below.

[0101] Users can use devices such as mobile phones and tablets to shoot videos and capture every moment of their lives. Currently, devices support shooting in Movie mode, which provides multiple LUTs for achieving different cinematic looks. When users shoot in Movie mode, the device uses the LUTs to process the captured video image, resulting in a professional, cinematic look.

[0102] The following combination Figure 3 and Figure 4 Let's illustrate the video shooting process in movie mode.

[0103] Take the mobile phone as an example, Figure 3 As shown in Figure (a), when a user needs to use a mobile phone to shoot a video, the user operates (such as clicking) the icon 301 of the camera application in the main screen interface of the mobile phone, and then the mobile phone displays the following Figure 3 The interface 302 shown in FIG. (b) is a preview interface of the photo mode. The working mode control 3021 in the interface 302 may include: portrait mode, photo mode, video mode, movie mode and professional mode. In response to the user selecting the movie mode 303, the mobile phone displays the following Figure 3 The interface 304 shown in Figure (c) is a preview interface before recording in movie mode. The mobile phone can display a prompt message 305 in the interface 304: "Landscape mode for a more cinematic feel", which is used to prompt the user to adjust the phone to landscape mode before recording the video. Afterwards, when the user adjusts the phone to landscape mode, the phone displays the following Figure 3 Interface 304, shown in Figure (d), is a preview interface for recording video in landscape mode. Interface 304 may include a capture control 306, a LUT control 307, and a settings control 308. Of course, interface 304 may also include other controls, such as a flash control. Capture control 306 is a virtual shutter button; the user triggers the phone to start recording by operating it. LUT control 307 is used to enable the LUT function. Settings control 308 is used to implement capture-related settings.

[0104] In response to the user's operation on the LUT control 307, such as Figure 4As shown, the mobile phone turns on the LUT function and displays the LUT bar 309 on the interface 304. The LUT bar 309 includes an icon of each global LUT in the multiple global LUTs pre-set by the system for the user to select. In some embodiments, the user can manually operate (such as clicking to select) a global LUT icon in the LUT bar 309, and then the user operates the shooting control 306 to use the filter effect corresponding to this global LUT to shoot the video in movie mode, that is, use this global LUT to process the captured video image, and the video thus shot will present the filter effect corresponding to this global LUT. In other embodiments, the mobile phone can automatically select a global LUT from the multiple pre-set global LUTs based on the picture content in the preview interface before recording displayed on the interface 304. At this time, the user operates the shooting control 306 to use the filter effect corresponding to this global LUT to shoot the video in movie mode, that is, use this global LUT to process the captured video image, and the video thus shot will present the filter effect corresponding to this global LUT.

[0105] The global LUT is used to process the entire captured video image. However, due to the influence of the light in the shooting environment, the video image processed with the global LUT may have color cast problems. For example, when shooting a person in movie mode, if the light in the shooting environment is natural light, the skin color of the portrait in the video image processed with the global LUT is normal. However, when the light in the shooting environment is low color temperature night scene neon light, using the same global LUT to process the video image will cause the skin color of the portrait in the processed video image to have a greenish cast. Skin color is a color that the human eye is very sensitive to. If the skin color is cast, it will greatly affect the user's perception and reduce the user experience.

[0106] However, since the global LUT is effective for the entire video image, if the global LUT is adjusted to correct the problem of skin color cast, it will cause color cast when shooting again in a shooting environment where the skin color was originally normal. In addition, colors in the video image that are close to skin color (such as red, orange, and yellow) will be affected and cause color cast.

[0107] To this end, an embodiment of the present application provides a video processing method that can use a global LUT to process the entire video image, and use a correction LUT corresponding to the light source properties of the video image to process the area where a preset target is located in the video image. The correction LUT can correct the color of the preset target under the illumination of a light source with the light source properties. Therefore, the video image processed by the global LUT and the correction LUT can not only have the filter effect corresponding to the global LUT, but also effectively avoid the color cast problem of the preset target.

[0108] The video processing method provided in the embodiment of the present application is explained in detail below.

[0109] Figure 5 This is a flow chart of a video processing method provided by an embodiment of the present application. The method is applied to a terminal, which may be the above-mentioned terminal. Figure 1-Figure 2 The terminal described in the embodiment. Figure 5 , the method includes the following steps.

[0110] Step 501: The terminal obtains multiple frames of video images in a video.

[0111] For example, the video format can be LOG format, which is a video recorded using a LOG logarithmic function and is a grayscale material video. Of course, the video format can also be other formats, which is not limited in this embodiment of the present application.

[0112] The multiple frames of video images may be continuous video images in the video, in which case the terminal obtains each frame of the video. Alternatively, the multiple frames of video images may be video images separated by a certain number of frames in the video, in which case the terminal obtains one frame of video image from the video after a certain number of frames have passed.

[0113] Optionally, the operation of the terminal acquiring multiple frames of video images in a video may be implemented in the following two possible ways.

[0114] In a first possible manner, the terminal may obtain multiple frames of video images in a video being shot by a camera during the shooting process of the camera.

[0115] For example, the terminal can obtain multiple frames of video images from the video being recorded by the camera during the process of recording video through the camera in movie mode.

[0116] In this manner, the terminal processes the video being captured by the camera in real time. Specifically, each time the terminal obtains a frame of video image from the video being captured by the camera, it processes the frame of video image according to the following steps 502 to 505. In this way, when the camera finishes shooting, the processing of multiple frames of video images in the video captured by the camera can be completed to obtain a processed video, which in the embodiment of the present application is a video with the filter effect corresponding to the LUT.

[0117] In a second possible manner, the terminal may obtain multiple frames of video images in the video captured by the camera after the camera finishes shooting.

[0118] For example, after the camera finishes recording, the terminal can obtain multiple frames of video images in the video recorded by the camera.

[0119] In this method, the terminal performs post-processing on the video captured by the camera. Specifically, for each frame of video image obtained by the terminal from the video captured by the camera, the terminal can process the frame of video image according to the following steps 502-505. In this way, after completing the processing of multiple frames of video images in the video, a processed video can be obtained. In the embodiment of the present application, a video with the filter effect corresponding to the LUT is obtained.

[0120] Step 502: For any one frame of the multiple frames of video images, the terminal performs target detection on the frame of video image.

[0121] After the terminal performs target detection on this frame of video image, it can detect whether there is a preset target in this frame of video image. The preset target can be set in advance, such as a portrait, sky, water surface, animal, plant, building, etc., which is not limited in this embodiment of the present application. It is worth noting that in the embodiment of the present application, the target can include not only large categories such as portraits, sky, water surface, animals, plants, and buildings, but also small categories under each large category. For example, for animals, it can include small categories such as cats and dogs.

[0122] Optionally, the terminal may use a region of interest (ROI) detection algorithm to perform target detection on the frame of video image, thereby detecting the subject in the region of interest in the frame of video image. Alternatively, the terminal may use a salient target detection algorithm to perform target detection on the frame of video image, thereby detecting the subject in the salient region in the frame of video image. Of course, the terminal may also perform target detection on the frame of video image in other ways, which is not limited in the embodiments of the present application.

[0123] The process of target detection by the terminal for this frame of video image is the process of semantic segmentation of this frame of video image, thereby identifying the semantics of each area in this frame of video image (i.e., the area where each target is located), that is, determining the category and area of ​​each target in this frame of video image. In this way, the terminal can determine whether there is a preset target in this frame of video image, and if so, determine the area where the preset target is located in this frame of video image.

[0124] It's worth noting that the terminal's target detection for this frame of video is primarily intended to determine the area within the frame where the preset target exists, if one exists within the frame. Therefore, to reduce processing pressure and optimize processing performance, the terminal can use the low-resolution image corresponding to the frame of video to perform target detection. If the preset target exists in the low-resolution image, then the preset target also exists in the frame of video, and the location of the preset target in the low-resolution image is the same as the location of the preset target in the frame of video.

[0125] For example, the video format captured by the camera is in LOG format, and the low-resolution format corresponding to the LOG format may be Rec.709 format. In this case, the terminal can convert the LOG format video captured by the camera to obtain a Rec.709 format video, and then use the Rec.709 format video to perform target detection. In other words, target detection is performed on multiple frames of video images in the Rec.709 format video, that is, the video images in the Rec.709 format video are used as input data for the target detection algorithm. The output data of the target detection algorithm is the location of the area where the preset target exists in the video image in the Rec.709 format video, that is, the location of the area where the preset target exists in the video image in the LOG format video.

[0126] Another point worth noting is that after the terminal performs target detection on this frame of video image, if the preset target exists in this frame of video image, the terminal can obtain a mask image of the preset target in this frame of video image. This mask image can indicate the location of the preset target in the area of ​​the video image. Using this mask image, the area where the preset target is located can be segmented from this frame of video image.

[0127] For example, the preset target is a portrait, and this frame of video image is Figure 6 The terminal detects the target in this frame of video image and obtains the following: Figure 6 The target detection result shown in Figure (b) in the figure can be a mask image. The white part in the mask image is used to indicate the position of the area where the portrait is located in this frame of video image, and the black part in the mask image is used to indicate the position of other areas in this frame of video image except the area where the portrait is located. The mask image can be used to segment the area where the portrait is located from this frame of video image, that is, the mask image can be used to block out other areas in this frame of video image except the area where the portrait is located, and retain the area where the portrait is located in this frame of video image, so as to obtain the following: Figure 6 The image shown in Figure (c).

[0128] Specifically, the mask image can be an image whose pixel values ​​include 0 and 255, such as Figure 6 As shown in Figure (b), the white part in the mask image is the part with a pixel value of 255, and the black part in the mask image is the part with a pixel value of 0. In this case, the mask image is used to segment the area where the portrait is located from this frame of video image, and each pixel value in the mask image is ANDed with the pixel value at the corresponding position in this frame of video image. The ANDed operation means that when a pixel value in the mask image is 0, the pixel value at the corresponding position is set to 0, and when a pixel value in the mask image is not 0 (that is, 255), the pixel value at the corresponding position in this frame of video image is retained. In this way, after performing the ANDed operation on each pixel value in the mask image and the pixel value at the corresponding position in this frame of video image, the pixel value in the area where the portrait is located in this frame of video image remains unchanged, and the pixel values ​​in other areas of this frame of video image except the area where the portrait is located are all 0, that is, we get Figure 6 The image shown in Figure (c) is an image segmented from this frame of video image, which blocks other areas (that is, other areas are black) and retains the area where the portrait is located.

[0129] Step 503: If a preset target exists in this frame of video image, the terminal obtains the light source properties of this frame of video image.

[0130] The light source attributes of a video frame refer to the relevant attributes of the light source in the shooting environment of the video frame. The light source illuminating the preset target can be determined based on the light source attributes. For example, the light source attributes may include one or more of brightness, light source type, and correlated color temperature (CCT), etc., which are not limited in this embodiment of the present application.

[0131] Optionally, the terminal may use a software algorithm or a hardware device to obtain the light source properties of the video frame. The terminal may use a software algorithm to obtain the light source properties of the video frame in any of the following possible methods: the first method to the fourth method. The terminal may use a hardware device to obtain the light source properties of the video frame in the following possible method: the fifth method.

[0132] In a first possible manner, the terminal may use an automatic white balance (AWB) algorithm to estimate the color temperature of the entire frame of video image to obtain the light source properties of the frame of video image.

[0133] For example, the terminal can use the AWB algorithm to determine the color temperature of each pixel in this frame of video image. Afterwards, the terminal can use the average value of the color temperature of all pixels in this frame of video image as the light source attribute of this frame of video image; or, perform weighted averaging on the color temperature of all pixels in this frame of video image to obtain the light source attribute of this frame of video image. Among them, the weight of the pixel points in the area where the preset target is located in this frame of video image is greater than the weight of the pixel points in other areas of this frame of video image except the area where the preset target is located, the weight of each pixel point in the area where the preset target is located is the same, the weight of each pixel point in other areas except the area where the preset target is located is the same, and the weight of the pixel points in the area where the preset target is located and the weight of the pixel points in other areas except the area where the preset target is located can be set in advance.

[0134] In a second possible manner, the terminal may use an AWB algorithm to estimate the color temperature of the area where the preset target is located in the frame of video image, and obtain the light source properties of the frame of video image.

[0135] Since the primary purpose of the present embodiment is to determine the light source illuminating a preset target, if the light in the shooting environment is mixed light, more accurate light source properties can be obtained by estimating the color temperature of only the area where the preset target is located in the frame of video image. For example, the terminal can use an AWB algorithm to determine the color temperature of each pixel in the area where the preset target is located in the frame of video image, and then use the average color temperature of all pixels in the area where the preset target is located in the frame of video image as the light source property of the frame of video image.

[0136] In a third possible manner, the terminal may use an AWB algorithm to estimate the color temperature of neutral gray pixels in the frame of video image to obtain the light source properties of the frame of video image.

[0137] Neutral gray pixels, also called neutral gray blocks, are encoded as Block 22. Neutral gray pixels are pixels where the red, green, and blue values ​​in the RGB value are equal. For example, the terminal can use the AWB algorithm to determine the color temperature of each neutral gray pixel in a video frame and then use the average color temperature of all neutral gray pixels in the video frame as the light source attribute for the video frame.

[0138] In a fourth possible manner, the terminal may use an AWB algorithm to estimate the color temperature of the area where the preset target is located and the neutral gray pixel points in the video image frame to obtain the light source properties of the video image frame.

[0139] For example, the terminal can use the AWB algorithm to determine the color temperature of each pixel in the area where the preset target is located in this frame of video image and the color temperature of each neutral gray pixel in this frame of video image. Afterwards, the terminal can use the average value of the color temperature of all pixels in the area where the preset target is located in this frame of video image and the color temperature of all neutral gray pixels in this frame of video image as the light source attribute of this frame of video image; or, perform weighted averaging on the color temperature of all pixels in the area where the preset target is located in this frame of video image and the color temperature of all neutral gray pixels in this frame of video image to obtain the light source attribute of this frame of video image. Among them, the weight of the pixel in the area where the preset target is located in this frame of video image is greater than the weight of the neutral gray pixel in this frame of video image, the weight of each pixel in the area where the preset target is located is the same, the weight of each neutral gray pixel is the same, and the weight of the pixel in the area where the preset target is located and the weight of the neutral gray pixel can be set in advance.

[0140] In a fifth possible manner, a spectrum detection sensor may be provided in the terminal, and the terminal may detect the light source properties of the frame of video image through the spectrum detection sensor.

[0141] For example, a spectral detection sensor may be provided in the terminal, and the spectral detection sensor includes m×n detection elements. In this way, the spectral detection sensor can divide the shooting field of view (FOV) into m×n areas to respectively detect the spectral information (such as color temperature) of each area in the m×n areas. Wherein, m and n are both positive integers.

[0142] The terminal can use the spectrum detection sensor to detect the color temperature of each of the m×n regions in the video frame, and then determine the light source attributes of the video frame based on the color temperature of each of the m×n regions in the video frame. For example, the terminal can determine the light source attributes of the video frame as the average of the color temperatures of the m×n regions in the video frame. Alternatively, the terminal can take a weighted average of the color temperatures of the regions containing the preset target and the color temperatures of the regions not containing the preset target in the video frame to obtain the light source attributes of the video frame. Alternatively, the terminal can determine the light source attributes of the video frame as the average of the color temperatures of the regions containing the preset target in the video frame. Alternatively, the terminal can determine the light source attributes of the video frame as the average of the color temperatures of the regions containing neutral gray pixels in the video frame. Alternatively, the terminal can determine the light source attributes of the video frame as the average of the color temperatures of the regions containing the preset target and the regions containing neutral gray pixels in the video frame. Alternatively, the terminal may perform a weighted average of the color temperatures of the m×n regions containing the preset target and the color temperatures of the regions containing neutral gray pixels in the frame of video image to obtain the light source attributes of the frame of video image. The weight of the region containing the preset target among the m×n regions in the frame of video image is greater than the weight of the region not containing the preset target, and the weight of the region containing the preset target among the m×n regions in the frame of video image is greater than the weight of the region containing neutral gray pixels. The weights of the regions containing the preset target are the same, the weights of the regions not containing the preset target are the same, and the weights of the regions containing neutral gray pixels are the same. The weights of the regions containing the preset target, the weights of the regions not containing the preset target, and the weights of the regions containing neutral gray pixels can all be set in advance.

[0143] For example, the preset target is a portrait. The terminal is provided with a spectrum detection sensor, which includes 6×6 detection elements, that is, 36 detection elements are provided. Figure 7 As shown, these 36 detection elements can be used to detect the color temperature of the 36 areas in this frame of video image. Among the 36 areas in this frame of video image, Area A, Area B, Area C, Area D, Area E, Area F, Area G, Area H, Area I, Area J, Area K, and Area L contain human portraits. In this case, the terminal can determine the average color temperature of these 12 areas as the light source attribute of this frame of video image.

[0144] It is worth noting that if the terminal obtains the color temperature in the light source attributes of this frame of video image, the terminal can also obtain other attributes such as the light source type and brightness corresponding to the color temperature, and the terminal can also use these attributes as the light source attributes of this frame of video image.

[0145] Step 504: The terminal obtains a correction LUT of the frame of video image according to the light source properties of the frame of video image.

[0146] The correction LUT for this frame of video image is a LUT used to perform color correction on the area where the preset target is located in this frame of video image. The correction LUT for this frame of video image is used to correct the color of the preset target under the illumination of a light source having the light source properties of this frame of video image. That is, the correction LUT for this frame of video image is used to correct the color of the preset target under the illumination of the light source in the shooting environment of this frame of video image. In this case, after using the correction LUT for this frame of video image to process the area where the preset target is located in this frame of video image, the color cast problem in the area where the preset target is located in this frame of video image can be improved. For example, the correction LUT can be a 3D LUT.

[0147] Optionally, the terminal may obtain the correction LUT of the frame of video image according to the light source attribute of the frame of video image in the following two possible ways.

[0148] In the first possible method, the terminal can obtain the correction LUT corresponding to the light source attributes of this frame of video image from the correspondence between the light source attributes and the correction LUT. The correction LUT corresponding to the light source attributes of this frame of video image is the correction LUT of this frame of video image.

[0149] The terminal can pre-store a mapping between light source attributes and correction LUTs. This mapping includes multiple light source attributes and multiple correction LUTs. The correction LUT corresponding to each light source attribute can be used to correct the color of a preset target under a light source with that attribute. This mapping can be set by technical personnel based on actual needs and experience, such as the user's visual experience requirements.

[0150] For example, if the light source attributes of this frame of video image are "Brightness: Medium, Light Source Type: TL84, Color Temperature: 4000K (Kelvin temperature unit)", the terminal can obtain the correction LUT corresponding to the light source attributes of this frame of video image from the correspondence between the light source attributes and the correction LUT shown in Table 2 below, which is "Correction LUT for Indoor Incandescent Light", and use the obtained correction LUT "Correction LUT for Indoor Incandescent Light" as the correction LUT for this frame of video image.

[0151] Table 2

[0152]

[0153] In the embodiment of the present application, only Table 2 is used as an example to illustrate the correspondence between light source properties and the correction LUT. Table 2 does not limit the embodiment of the present application.

[0154] It should be noted that if the terminal does not obtain the correction LUT corresponding to the light source attributes of this frame of video image from the correspondence between the light source attributes and the correction LUT, that is, the correction LUT corresponding to the light source attributes of this frame of video image does not exist in the correspondence, then the terminal can determine two light source attributes from the correspondence, and the light source attributes of this frame of video image are between these two light source attributes. Then, the terminal can obtain the correction LUT corresponding to each of the two light source attributes from the correspondence, and then perform interpolation operations (including but not limited to linear interpolation operations, etc.) on the two obtained correction LUTs to obtain the correction LUT of this frame of video image.

[0155] For example, the light source attribute of this frame of video image is "color temperature: 4300K". The terminal does not obtain the correction LUT corresponding to the light source attribute of this frame of video image from the correspondence between the light source attribute and the correction LUT shown in Table 2 above. That is, the correction LUT corresponding to the light source attribute of this frame of video image does not exist in the correspondence. In this case, since the light source attribute "color temperature: 4300K" of this frame of video image is between the light source attribute "color temperature: 5000K and above" and the light source attribute "color temperature: 4000K" in the corresponding relationship, the terminal can obtain the correction LUT corresponding to the light source attribute "color temperature: 5000K and above" from the corresponding relationship as the "correction LUT for natural light", and obtain the correction LUT corresponding to the light source attribute "color temperature: 4000K" from the corresponding relationship as the "correction LUT for indoor incandescent light". After that, the terminal can perform interpolation operations on the correction LUT "correction LUT for natural light" and the correction LUT "correction LUT for indoor incandescent light" to obtain the correction LUT for this frame of video image.

[0156] In a second possible approach, the terminal obtains the correction LUT for each of the k frames preceding the current frame in the multi-frame video image. Based on the correspondence between light source attributes and correction LUTs, the terminal obtains the correction LUT corresponding to the light source attributes of the current frame. The terminal then interpolates the correction LUTs for all of the k frames with the correction LUT corresponding to the light source attributes of the current frame to obtain the correction LUT for the current frame.

[0157] k is a positive integer, and k can be set in advance, such as k can be 1, 2, 3, etc., which is not limited in the embodiment of the present application.

[0158] Since the terminal processes each video image it acquires, and during this processing, it acquires the correction LUT of the processed video image, when the terminal acquires a certain video image, it has already processed the video image before this frame of video image, and thus the terminal can now acquire the correction LUT of these preceding video images. Afterwards, the terminal can perform interpolation operations (including but not limited to linear interpolation operations, etc.) on the correction LUTs of these preceding video images with the correction LUT corresponding to the light source properties of this frame of video image to obtain the correction LUT of this frame of video image. In this way, the smoothness of the filter effect changes after the area where the preset target in this frame of video image is located is processed using the correction LUT of this frame of video image, that is, the time domain smoothness of the color correction effect of the preset target is guaranteed.

[0159] It should be noted that if the terminal does not obtain the correction LUT corresponding to the light source attribute of this frame of video image from the correspondence between the light source attribute and the correction LUT, that is, the correction LUT corresponding to the light source attribute of this frame of video image does not exist in the correspondence, then the terminal can determine two light source attributes from the correspondence, and the light source attribute of this frame of video image is between these two light source attributes. Then, the terminal can obtain the correction LUT corresponding to each of the two light source attributes from the correspondence, and then perform interpolation operation on the two obtained correction LUTs to obtain the correction LUT corresponding to the light source attribute of this frame of video image. Afterwards, the terminal performs interpolation operation on the correction LUT of all video images in the k frames of video image and the correction LUT corresponding to the light source attribute of this frame of video image to obtain the correction LUT of this frame of video image.

[0160] It is worth noting that before processing the video, that is, before step 501, the terminal has already determined the global LUT for processing the entire video image. Optionally, the global LUT can be manually selected by the user, or it can be automatically selected by the terminal based on the image content displayed in the camera preview interface. Of course, it can also be selected by other means, which is not limited in this embodiment of the present application. For example, the global LUT can be a 3D LUT.

[0161] For example, Figure 3 As shown in FIG. 3 (d), the terminal displays an interface 304 before recording in movie mode. The interface 304 may include a shooting control 306 and a LUT control 307. After the user operates the LUT control 307, the terminal responds to the operation of the LUT control 307, as shown in FIG. Figure 4As shown, the terminal enables the LUT function and displays a LUT bar 309 on interface 304. LUT bar 309 includes an icon for each of the multiple pre-set global LUTs. The user then manually selects a global LUT icon in LUT bar 309, or the terminal automatically selects a global LUT icon in LUT bar 309 based on the content of the preview interface before recording displayed on interface 304. If the user then operates the capture control 306, the terminal responds to the operation of the capture control 306 by capturing a video through the camera and acquiring multiple frames of video images from the video being captured by the camera. The terminal can then use the selected global LUT to process the acquired video images.

[0162] Step 505: The terminal processes the frame of video image using the global LUT and the correction LUT of the frame of video image to obtain a processed video image.

[0163] The global LUT is a LUT used to process the entire frame of video image, and the correction LUT of the frame of video image is a LUT used to process the area where a preset target is located in the frame of video image.

[0164] Optionally, the terminal uses the global LUT and the correction LUT of the frame of video image to process the frame of video image to obtain the processed video image in the following two possible ways.

[0165] In the first possible method, the terminal uses the global LUT to process the entire frame of video image to obtain a first image, and uses the correction LUT of the frame of video image to process the area where the preset target is located in the frame of video image to obtain a second image. Thereafter, the first image and the second image are fused to obtain a processed video image.

[0166] This method processes a single frame of video in parallel. In this case, a dual video container can be used to store two video images, both representing the same frame. The global LUT is then used to process one video image as a whole, while the correction LUT is used to process the preset target area in the other video image. The processed results of the two video images are then fused to produce the final processed video image.

[0167] The terminal uses the global LUT to process the entire frame of video image, that is, to process the entire frame of video image into an image with the filter effect of the global LUT. The process of the terminal using the global LUT to process the entire frame of video image is similar to the operation of using a certain LUT to process the entire image in the related art, and this embodiment of the application will not be elaborated in detail.

[0168] As an example, the terminal uses the correction LUT of this frame of video image to process the area where the preset target is located in this frame of video image. The operation can be: use the correction LUT of this frame of video image to process all pixel points in this frame of video image, and then segment the area where the preset target is located from the processed image to obtain a second image.

[0169] For example, the default target is a portrait. Figure 6 The video image shown in Figure (a) contains a portrait. The terminal can use the correction LUT of the video image to correct the Figure 6 All the pixels in the video image shown in Figure (a) are processed, and then the area where the portrait is located is segmented from the processed image to obtain a second image. When segmenting the area where the portrait is located from the processed image, the image can be segmented using Figure 6 The mask image of the portrait in the video frame shown in Figure (b) is used to segment the region where the portrait is located from the processed image to obtain a second image. Specifically, each pixel value in the mask image can be ANDed with the pixel value at the corresponding position in the processed image. In this case, the pixel value of the region where the portrait is located in the processed image remains unchanged, and the pixel values ​​of all other regions in the processed image except the region where the portrait is located are all 0, thus obtaining a second image. The second image is an image segmented from the processed image, with other regions obscured (i.e., other regions are black) and the region where the portrait is located retained.

[0170] As another example, the operation of the terminal using the correction LUT of this frame of video image to process the area where the preset target is located in this frame of video image can be: using the correction LUT of this frame of video image to process all pixel points in the area where the preset target is located that is segmented during target detection to obtain a second image.

[0171] For example, the default target is a portrait. Figure 6 The video image shown in Figure (a) shows the area where the preset target is segmented during target detection. Figure 6 As shown in Figure (c), Figure 6 The image shown in Figure (c) is an image segmented from the video image, which blocks other areas (i.e., other areas are black) and retains the area where the portrait is located. The terminal can use the correction LUT of the video image to Figure 6 All pixels in the image shown in Figure (c) are processed to obtain a second image.

[0172] As an example, the operation of the terminal to fuse the first image with the second image can be: according to the position of the area where the preset target is located in this frame of video image, the area where the preset target is located in the second image is fused with other areas in the first image except the area where the preset target is located.

[0173] For example, during image fusion, for each pixel position in the first image, if this pixel position belongs to an area other than the area where the preset target is located in the first image, the pixel value of this pixel position in the first image is used as the pixel value of the corresponding pixel position in the processed video image; if this pixel position belongs to the area where the preset target is located in the first image, the pixel value of this pixel position in the second image is used as the pixel value of the corresponding pixel position in the processed video image. In this way, the areas other than the area where the preset target is located in the first image remain unchanged, and the area where the preset target is located in the second image is fused to obtain the processed video image.

[0174] Optionally, in order to ensure a smooth transition in color tone between the area where the preset target is located and other areas, alpha blending processing can also be performed on the edge of the area where the preset target is located during image fusion. Specifically, when performing image fusion, for each pixel position in the first image, if this pixel position belongs to an area other than the area where the preset target is located in the first image, the pixel value of this pixel position in the first image is used as the pixel value of the corresponding pixel position in the processed video image; if this pixel position is located at the edge of the area where the preset target is located in the first image, then the pixel value of this pixel position in the first image and the pixel value of this pixel position in the second image are weighted averaged according to the weight of the first image and the weight of the second image to obtain the pixel value of the corresponding pixel position in the processed video image; if this pixel position is located inside the area where the preset target is located in the first image, then the pixel value of this pixel position in the second image is used as the pixel value of the corresponding pixel position in the processed video image.

[0175] The weight of the first image and the weight of the second image can be set in advance, and the sum of the weight of the first image and the weight of the second image is 1. For example, the weight of the first image can be greater than or equal to 20% and less than 100%, and the weight of the second image can be greater than 0% and less than or equal to 80%.

[0176] In this case, when the terminal performs weighted averaging of the pixel value of the pixel position in the first image and the pixel value of the pixel position in the second image based on the weight of the first image and the weight of the second image, the terminal can multiply the pixel value of the pixel position in the first image by the weight of the first image to obtain the first pixel value, multiply the pixel value of the pixel position in the second image by the weight of the second image to obtain the second pixel value, and add the first pixel value to the second pixel value to obtain the pixel value of the corresponding pixel position in the processed video image.

[0177] As another example, the terminal may obtain the global LUT weight and the correction LUT weight in advance, for example, the global LUT weight and the correction LUT weight may be obtained before step 501. In this case, the terminal may perform image fusion on the first image and the second image by: fusing the first image and the second image according to the global LUT weight and the correction LUT weight to obtain a processed video image.

[0178] The global LUT weight is used to indicate the weight of the pixel values ​​of the pixels processed by the global LUT when performing image fusion. That is, the global LUT weight is used to indicate the weight of the pixel values ​​of the pixels in the area where the preset target is located in the first image when performing image fusion. The correction LUT weight is used to indicate the weight of the pixel values ​​of the pixels processed by the correction LUT when performing image fusion. That is, the correction LUT weight is used to indicate the weight of the pixel values ​​of the pixels in the area where the preset target is located in the second image when performing image fusion. The sum of the global LUT weight and the correction LUT weight is 1.

[0179] The terminal may obtain the global LUT weight and the correction LUT weight by: displaying a slider bar indicating the correction LUT weight; if a sliding operation of a slider in the slider bar is detected, determining the weight corresponding to the position of the slider in the slider bar as the correction LUT weight; and subtracting the correction LUT weight from 1 to obtain the global LUT weight. Alternatively, the terminal may obtain the global LUT weight corresponding to the global LUT from the corresponding relationship between the global LUT and the global LUT weight, and subtracting the global LUT weight from 1 to obtain the correction LUT weight.

[0180] The slider is a slider control. Users can change the position of the slider by sliding the slider on the slider, and then change the correction LUT weight. In other words, users can manually set the correction LUT weight. For example, Figure 8As shown, before the camera captures a video, the terminal displays a pre-recording interface 304 in movie mode. Interface 304 may include a slider 310. The user can slide the slider in slider 310. In response to the user sliding the slider in slider 310, the terminal determines that the weight corresponding to the position of the slider in slider 310 is 80%. The terminal may then determine that the correction LUT weight is 80% and the global LUT weight is 20%.

[0181] Because some global LUTs have strong color styles, to ensure a harmonious overall color style for the video image, the pixel values ​​processed by the global LUT need to be weighted to a certain extent during image fusion. In this case, the terminal can automatically determine the global LUT weight and the correction LUT weight. In other words, the terminal can directly obtain the global LUT weight corresponding to the global LUT from the corresponding relationship between the global LUT and the global LUT weight, and calculate the correction LUT weight accordingly.

[0182] The terminal can pre-store a correspondence between a global LUT and a global LUT weight. This correspondence includes multiple global LUTs and multiple global LUT weights. The global LUT weight corresponding to each global LUT is the weight that the pixel value of the pixel processed by this global LUT should occupy during image fusion. This correspondence can be set by technical personnel based on actual needs and experience, such as the user's visual experience requirements.

[0183] For example, if the global LUT is "global LUT1", the terminal can obtain the global LUT weight corresponding to the global LUT as 30% from the correspondence between the global LUT and the global LUT weight shown in Table 3 below, and then determine that the global LUT weight is 30% and the correction LUT weight is 70%.

[0184] Table 3

[0185] Global LUT Global LUT Weights Global LUT1 30％ Global LUT2 20％ Global LUT3 10％ …… ……

[0186] In the embodiment of the present application, only the global LUT and the global LUT weight are illustrated by taking Table 3 above as an example, and Table 3 above does not limit the embodiment of the present application.

[0187] The terminal performs image fusion on the first image and the second image according to the global LUT weight and the correction LUT weight to obtain the processed video image. The operation can be: according to the global LUT weight and the correction LUT weight, and according to the position of the area where the preset target is located in this frame of video image, the area where the preset target is located in the second image is fused with the first image.

[0188] For example, during image fusion, for each pixel position in the first image, if this pixel position belongs to an area other than the area where the preset target is located in the first image, the pixel value of this pixel position in the first image is used as the pixel value of the corresponding pixel position in the processed video image; if this pixel position belongs to the area where the preset target is located in the first image, then according to the global LUT weight and the correction LUT weight, the pixel value of this pixel position in the first image and the pixel value of this pixel position in the second image are weighted averaged to obtain the pixel value of the corresponding pixel position in the processed video image. In this way, the areas other than the area where the preset target is located in the first image are kept unchanged, and the area where the preset target is located in the first image and the area where the preset target is located in the second image are fused according to the global LUT weight and the correction LUT weight to obtain the processed video image, thereby ensuring that the color style of the area where the preset target is located in the processed video image is relatively consistent with that of other areas.

[0189] Optionally, in order to ensure a smooth transition in color tone between the area where the preset target is located and other areas, transparency blending processing can also be performed on the edge of the area where the preset target is located during image fusion. Specifically, when performing image fusion, for each pixel position in the first image, if this pixel position belongs to an area other than the area where the preset target is located in the first image, the pixel value of this pixel position in the first image is used as the pixel value of the corresponding pixel position in the processed video image; if this pixel position is at the edge of the area where the preset target is located in the first image, then according to the weight of the first image and the weight of the second image, the pixel value of this pixel position in the first image and the pixel value of this pixel position in the second image are weighted averaged to obtain the pixel value of the corresponding pixel position in the processed video image; if this pixel position is located inside the area where the preset target is located in the first image, then according to the global LUT weight and the correction LUT weight, the pixel value of this pixel position in the first image and the pixel value of this pixel position in the second image are weighted averaged to obtain the pixel value of the corresponding pixel position in the processed video image.

[0190] In which, when the terminal performs weighted averaging of the pixel value of the pixel position in the first image and the pixel value of the pixel position in the second image based on the global LUT weight and the correction LUT weight, the terminal multiplies the pixel value of the pixel position in the first image by the global LUT weight to obtain a third pixel value, and multiplies the pixel value of the pixel position in the second image by the correction LUT weight to obtain a fourth pixel value, and then adds the third pixel value to the fourth pixel value to obtain the pixel value of the corresponding pixel position in the processed video image.

[0191] The following combination Figure 9The parallel processing process of the video images in the first possible manner is described below.

[0192] like Figure 9 As shown, the global LUT is used to process all pixels in this frame of video image to obtain the first image. The correction LUT of this frame of video image is used to process all pixels in this frame of video image, and then the mask image of the preset target in this frame of video image is used to segment the area where the preset target is located from the processed image to obtain the second image. Then, based on the global LUT weights and the correction LUT weights, as well as the position of the area where the preset target is located in this frame of video image, the area where the preset target is located in the second image is fused with the first image to obtain the processed video image.

[0193] In a second possible method, the terminal uses the global LUT to process the entire frame of video image to obtain a first image, and then uses the correction LUT of the frame of video image to process the area where the preset target is located in the first image to obtain a processed video image.

[0194] This method processes the video frame serially. In this case, the global LUT is first used to process the entire video frame, and then the correction LUT is used to process the preset target area in the processed image to obtain the processed video image.

[0195] Optionally, the terminal uses the correction LUT of this frame of video image to process the area where the preset target is located in the first image. The operation can be: segmenting the area where the preset target is located from the first image to obtain the preset target image, using the correction LUT of this frame of video image to process all pixel points in the preset target image to obtain a third image, and fusing the first image and the third image to obtain a processed video image.

[0196] When the terminal segments the area where the preset target is located from the first image, it can use the mask image of the preset target in this frame of video image to segment the area where the preset target is located from the first image, thereby obtaining a preset target image. Specifically, each pixel value in the mask image can be ANDed with the pixel value of the corresponding position in the first image. At this time, the pixel value of the area where the preset target is located in the first image remains unchanged, and the pixel values ​​of other areas in the first image except the area where the preset target is located are all 0, thus obtaining the preset target image. The preset target image is an image segmented from the first image that blocks other areas (i.e., other areas are black) and retains the area where the preset target is located.

[0197] As an example, the operation of the terminal to fuse the first image with the third image can be: according to the position of the area where the preset target is located in this frame of video image, the area where the preset target is located in the third image is fused with other areas in the first image except the area where the preset target is located.

[0198] For example, during image fusion, for each pixel position in the first image, if this pixel position belongs to an area other than the area where the preset target is located in the first image, the pixel value of this pixel position in the first image is used as the pixel value of the corresponding pixel position in the processed video image; if this pixel position belongs to the area where the preset target is located in the first image, the pixel value of this pixel position in the third image is used as the pixel value of the corresponding pixel position in the processed video image. In this way, the areas other than the area where the preset target is located in the first image remain unchanged, and the area where the preset target is located in the third image is fused to obtain the processed video image.

[0199] Optionally, in order to ensure a smooth transition in color tone between the area where the preset target is located and other areas, transparency blending processing can also be performed on the edge of the area where the preset target is located during image fusion. Specifically, when performing image fusion, for each pixel position in the first image, if this pixel position belongs to an area other than the area where the preset target is located in the first image, the pixel value of this pixel position in the first image is used as the pixel value of the corresponding pixel position in the processed video image; if this pixel position is located at the edge of the area where the preset target is located in the first image, then according to the weight of the first image and the weight of the third image, the pixel value of this pixel position in the first image and the pixel value of this pixel position in the third image are weighted averaged to obtain the pixel value of the corresponding pixel position in the processed video image; if this pixel position is located inside the area where the preset target is located in the first image, then the pixel value of this pixel position in the third image is used as the pixel value of the corresponding pixel position in the processed video image.

[0200] The weight of the first image and the weight of the third image may be pre-set, and the sum of the weight of the first image and the weight of the third image is 1. For example, the weight of the first image may be greater than or equal to 20% and less than 100%, and the weight of the third image may be greater than 0% and less than or equal to 80%.

[0201] In this case, when the terminal performs weighted averaging of the pixel value of the pixel position in the first image and the pixel value of the pixel position in the third image based on the weight of the first image and the weight of the third image, the terminal can multiply the pixel value of the pixel position in the first image by the weight of the first image to obtain a fifth pixel value, multiply the pixel value of the pixel position in the third image by the weight of the third image to obtain a sixth pixel value, and add the fifth pixel value to the sixth pixel value to obtain the pixel value of the corresponding pixel position in the processed video image.

[0202] As another example, the terminal may obtain the global LUT weight and the correction LUT weight in advance, for example, the global LUT weight and the correction LUT weight may be obtained before step 501. In this case, the terminal may perform image fusion on the first image and the third image by: fusing the first image and the third image according to the global LUT weight and the correction LUT weight to obtain a processed video image.

[0203] The terminal performs image fusion on the first image and the third image according to the global LUT weight and the correction LUT weight to obtain the processed video image. The operation can be: according to the global LUT weight and the correction LUT weight, and according to the position of the area where the preset target is located in this frame of video image, the area where the preset target is located in the third image is fused with the first image.

[0204] For example, during image fusion, for each pixel position in the first image, if this pixel position belongs to an area other than the area where the preset target is located in the first image, the pixel value of this pixel position in the first image is used as the pixel value of the corresponding pixel position in the processed video image; if this pixel position belongs to the area where the preset target is located in the first image, the pixel value of this pixel position in the first image and the pixel value of this pixel position in the third image are weighted averaged according to the global LUT weight and the correction LUT weight to obtain the pixel value of the corresponding pixel position in the processed video image. In this way, the areas other than the area where the preset target is located in the first image are kept unchanged, and the area where the preset target is located in the first image and the area where the preset target is located in the third image are fused according to the global LUT weight and the correction LUT weight to obtain the processed video image, thereby ensuring that the color style of the area where the preset target is located in the processed video image is relatively consistent with that of other areas.

[0205] Optionally, in order to ensure a smooth transition in tone between the area where the preset target is located and other areas, transparency blending processing can also be performed on the edge of the area where the preset target is located during image fusion. Specifically, when performing image fusion, for each pixel position in the first image, if this pixel position belongs to an area other than the area where the preset target is located in the first image, the pixel value of this pixel position in the first image is used as the pixel value of the corresponding pixel position in the processed video image; if this pixel position is located at the edge of the area where the preset target is located in the first image, then according to the weight of the first image and the weight of the third image, the pixel value of this pixel position in the first image and the pixel value of this pixel position in the third image are weighted averaged to obtain the pixel value of the corresponding pixel position in the processed video image; if this pixel position is located inside the area where the preset target is located in the first image, then according to the global LUT weight and the correction LUT weight, the pixel value of this pixel position in the first image and the pixel value of this pixel position in the third image are weighted averaged to obtain the pixel value of the corresponding pixel position in the processed video image.

[0206] Among them, when the terminal performs weighted averaging of the pixel value of the pixel position in the first image and the pixel value of the pixel position in the third image based on the global LUT weight and the correction LUT weight, the terminal multiplies the pixel value of the pixel position in the first image by the global LUT weight to obtain the seventh pixel value, and multiplies the pixel value of the pixel position in the third image by the correction LUT weight to obtain the eighth pixel value, and then adds the seventh pixel value to the eighth pixel value to obtain the pixel value of the corresponding pixel position in the processed video image.

[0207] The following combination Figure 10 The serial processing process of the video images in the second possible manner is described below.

[0208] like Figure 10 As shown, the global LUT is used to process all pixels in the video frame to obtain a first image. The mask image of the preset target in the video frame is used to segment the area where the preset target is located from the first image to obtain a preset target image. The correction LUT of the video frame is used to process all pixels in the preset target image to obtain a third image. The first image and the third image are fused to obtain a processed video image.

[0209] In an embodiment of the present application, multiple frames of video images are obtained in a video. For any one frame of the multiple frames of video images, when a preset target exists in this frame of video image, the light source properties of this frame of video image can be obtained, and the correction LUT of this frame of video image can be obtained based on the light source properties of this frame of video image. Afterwards, this frame of video image can be processed using the global LUT and the correction LUT. Since the correction LUT is used to perform color correction on the area where the preset target is located in this frame of video image, the video image processed by the global LUT and the correction LUT can not only have the filter effect corresponding to the global LUT, but also effectively avoid the color cast problem of the preset target.

[0210] It is worth noting that the preset target in the embodiment of the present application can be a target with a memory color, such as a portrait, and the memory color of the portrait is skin color. Of course, the preset target can also be other targets with memory colors, such as the sky, water, green plants, etc.

[0211] The following takes the preset target as an example, combined with Figure 11 Let's take an example to illustrate the above video processing method.

[0212] like Figure 11 As shown, the video processing method may include the following steps (1) to (5).

[0213] (1) The camera collects video stream.

[0214] The video stream captured by the camera may be a high-resolution video stream, such as a 4K resolution video stream. In this case, a low-resolution video stream corresponding to the high-resolution video stream may also be provided. The low-resolution video stream may be obtained by converting the high-resolution video stream, such as a video graphics array (VGA) resolution video stream.

[0215] The high-resolution video stream is used as the main channel for LUT processing, and the low-resolution video stream is used as the auxiliary channel to determine the mask map of the portrait. The mask map of the portrait is used to indicate the position of the portrait area in the video image.

[0216] (2) The video image in the high-resolution video stream is input into the image processing engine (IPE), which processes the high-resolution video image and outputs it to the LUT processing unit.

[0217] For example, the IPE may include multiple independent component analysis (ICA) units, local tone mapping (LTM) units, gamma correction units, etc., to process the video image.

[0218] Optionally, an electronic image stabilization (EIS) unit may be used to perform grid correction on the video image processed by the IPE to achieve motion compensation and thus achieve an image stabilization effect.

[0219] (3) Perform semantic segmentation on the video image in the low-resolution video stream to obtain a mask map of the portrait, and output the mask map of the portrait to the LUT processing unit.

[0220] Optionally, an artificial intelligence (AI) engine may be used to perform semantic segmentation on the video image in the low-resolution video stream to obtain a mask image of the portrait.

[0221] In this case, the EIS compensation grid of the main video image can also be synchronized to perform grid correction on the mask image of the portrait output by the AI ​​engine, and then the corrected mask image can be output to the LUT processing unit.

[0222] (4) Obtain the light source properties of the video image in the high-resolution video stream.

[0223] Optionally, the color temperature of each of the m×n regions in the video image may be detected by a spectrum detection sensor, and then the light source attribute of the video image may be determined according to the color temperature of each of the m×n regions.

[0224] For example, the terminal may perform a weighted average of the color temperatures of the m×n regions to obtain the light source attributes of the video image. In this case, the weight of the portrait region in the m×n regions may be greater than the weights of other regions in the m×n regions excluding the portrait region, and the weight of the face region in the portrait region may be greater than the weights of other regions in the portrait region excluding the face region.

[0225] (5) The LUT processing unit obtains a correction LUT according to the light source properties of the video image, uses the global LUT to process the high-resolution video image as a whole to obtain a first image, and uses the correction LUT to process all pixels in the high-resolution video image, and then uses the mask image of the portrait to segment the portrait area from the processed image to obtain a second image. Thereafter, the first image and the second image are fused to obtain a processed video image.

[0226] In this case, the color of the portrait area in the processed video image is corrected by the correction LUT, so the processed video image not only has the filter effect corresponding to the global LUT, but also effectively avoids the color cast problem of the portrait.

[0227] Optionally, the first image and the second image may be fused according to the global LUT weight and the correction LUT weight to obtain a processed video image.

[0228] For example, the global LUT weight and the correction LUT weight can be manually set by the user before shooting. For example, a slider bar can be displayed on the camera preview interface, and the user can slide the slider to change the position of the slider, thereby changing the correction LUT weight.

[0229] In some embodiments, before shooting, other sliders may be displayed on the camera preview interface. The user may change the position of the slider by sliding the slider on the other slider, and then change the degree of skin smoothing of the portrait. The higher the degree of skin smoothing, the smoother the skin of the portrait.

[0230] The following uses the light source attribute of a video image, which is the color temperature of the video image, as an example to exemplify the video processing method provided in the embodiment of the present application.

[0231] Figure 12 This is a flow chart of a video processing method provided by an embodiment of the present application. The method is applied to a terminal, which may be the above-mentioned terminal. Figure 1-Figure 2 The terminal described in the embodiment. Figure 12 , the method includes the following steps.

[0232] Step 1201: The terminal obtains multiple frames of video images in the video.

[0233] The operation of step 1201 is the same as above. Figure 5 The operation of step 501 in the embodiment is the same and will not be described in detail in the embodiment of the present application.

[0234] Step 1202: For any one frame of the multiple frames of video images, if a preset target exists in the frame of video image, the terminal obtains the color temperature of the frame of video image.

[0235] The terminal may first perform target detection on the frame of video image, and if a preset target is detected in the frame of video image, the color temperature of the frame of video image is obtained.

[0236] Among them, the terminal performs target detection on this frame of video image with the same operation as above. Figure 5The operation of step 502 in the embodiment is the same and will not be described in detail in the embodiment of the present application.

[0237] The operation of the terminal obtaining the color temperature of this frame of video image is the same as above. Figure 5 The operation of step 503 in the embodiment is the same and will not be described in detail in the embodiment of the present application.

[0238] In some possible cases, the color temperature of this frame of video image is the first color temperature, and the terminal then proceeds to step 1203. In other possible cases, the color temperature of this frame of video image is the second color temperature, and the terminal then proceeds to step 1204. The first color temperature is different from the second color temperature.

[0239] Step 1203: If the color temperature of the frame of video image is the first color temperature, the terminal uses the global LUT and the first correction LUT to process the frame of video image to obtain a processed video image.

[0240] It is worth noting that before processing the video, that is, before step 1201, the terminal has already determined in advance the global LUT for processing the entire video image. Optionally, the global LUT can be manually selected by the user, or it can be automatically selected by the terminal based on the image content displayed in the camera preview interface. Of course, it can also be selected in other ways, and the embodiments of the present application are not limited to this. For example, the global LUT can be a 3D LUT. The global LUT is a LUT used to process the entire frame of video image.

[0241] For example, if the terminal detects the selection operation of the movie mode, Figure 3 As shown in FIG. 3 (d), the terminal displays a camera preview interface 304 in movie mode. The camera preview interface 304 includes a LUT control 307. The LUT control 307 is used to start the LUT function. Afterwards, if the terminal detects a selection operation on the LUT control 307, Figure 4 As shown, the terminal displays a LUT bar 309 on the camera preview interface 304. The LUT bar 309 includes multiple LUT icons. These multiple LUT icons are icons of each global LUT in multiple global LUTs pre-set by the terminal. Subsequently, if the terminal detects a selection operation on one of the multiple LUT icons, it obtains the LUT corresponding to the LUT icon as the global LUT.

[0242] The first correction LUT is a LUT used to process the area containing the preset target in the current frame of video. The first correction LUT is used to correct the color of the preset target under illumination with a light source having a first color temperature. Thus, the first correction LUT can perform color correction on the area containing the preset target in the current frame of video. In this case, using the first correction LUT to process the area containing the preset target in the current frame of video can improve the color cast in the area containing the preset target in the current frame of video.

[0243] Before the terminal uses the global LUT and the first correction LUT to process the video image, the terminal can first obtain the first correction LUT according to the first color temperature. The operation of the terminal obtaining the first correction LUT according to the first color temperature is the same as the above. Figure 5 The operation of step 504 in the embodiment is similar and will not be elaborated in detail in the embodiment of the present application.

[0244] For example, the terminal may obtain the correction LUT corresponding to the first color temperature from the correspondence between the color temperature and the correction LUT, and the correction LUT corresponding to the first color temperature is the first correction LUT.

[0245] It should be noted that if the terminal does not obtain the correction LUT corresponding to the first color temperature from the correspondence between color temperature and correction LUT, that is, there is no correction LUT corresponding to the first color temperature in the correspondence, then the terminal can determine two color temperatures from the correspondence, and the first color temperature is between these two color temperatures. Then, the terminal can obtain the correction LUT corresponding to each of the two color temperatures from the correspondence, and then perform interpolation operations (including but not limited to linear interpolation operations, etc.) on the two obtained correction LUTs to obtain the first correction LUT.

[0246] For another example, the terminal can obtain the correction LUT used when processing each of the k frames preceding the current frame in the multi-frame video image, thereby obtaining k correction LUTs, where k is a positive integer. The terminal then obtains the correction LUT corresponding to the first color temperature based on the correspondence between color temperature and correction LUTs. The terminal then interpolates the k correction LUTs with the correction LUT corresponding to the first color temperature to obtain the first correction LUT.

[0247] It should be noted that if the terminal does not obtain the correction LUT corresponding to the first color temperature from the correspondence between color temperatures and correction LUTs, that is, if the correction LUT corresponding to the first color temperature does not exist in the correspondence, the terminal can determine two color temperatures from the correspondence, and the first color temperature is between the two color temperatures. The terminal can then obtain the correction LUT corresponding to each of the two color temperatures from the correspondence, and then interpolate the two obtained correction LUTs to obtain the correction LUT corresponding to the first color temperature. The terminal then interpolates the k correction LUTs and the correction LUT corresponding to the first color temperature to obtain the first correction LUT.

[0248] The terminal uses the global LUT and the first correction LUT to process this frame of video image, and the operation of obtaining the processed video image is the same as above. Figure 5 The operation of step 505 in the embodiment is the same and will not be described in detail in the embodiment of the present application.

[0249] Step 1204: If the color temperature of the frame of video image is the second color temperature, the terminal uses the global LUT and the second correction LUT to process the frame of video image to obtain a processed video image.

[0250] The second correction LUT is used to process the area containing the preset target in the current frame of video. The second correction LUT is used to correct the color of the preset target under the illumination of a light source having a second color temperature. Thus, the second correction LUT can perform color correction on the area containing the preset target in the current frame of video. In this case, using the second correction LUT to process the area containing the preset target in the current frame of video can improve the color cast in the area containing the preset target.

[0251] The second correction LUT is different from the first correction LUT. In the embodiment of the present application, when the color temperature of the video image is different, different correction LUTs can be used to perform color correction on the area where the preset target is located in the video image, thereby improving the flexibility and accuracy of color correction.

[0252] Among them, the operation of step 1204 is similar to the operation of step 1205, and will not be repeated in this embodiment of the application.

[0253] In an embodiment of the present application, multiple frames of video images in a video are obtained. For any frame of video images in the multiple frames of video images, the color temperature of this frame of video image can be obtained when a preset target exists in this frame of video image. Afterwards, if the color temperature of this frame of video is the first color temperature, the global LUT and the first correction LUT are used to process this frame of video image. If the color temperature of this frame of video is the second color temperature, the global LUT and the second correction LUT are used to process this frame of video image. In this way, when the color temperature of the video image is different, different correction LUTs can be used to perform color correction on the area where the preset target is located in the video image, so that the video image processed by the global LUT and the correction LUT can not only have the filter effect corresponding to the global LUT, but also effectively avoid the color cast problem of the preset target.

[0254] Figure 13 This is a schematic diagram of the structure of a video processing device provided by an embodiment of the present application. The device can be implemented as part or all of a computer device by software, hardware, or a combination of both. The computer device can be the above Figure 1-Figure 2 The terminal described in the embodiment. Figure 13 The device includes: a first acquisition module 1301, a second acquisition module 1302 and a processing module 1303.

[0255] The first acquisition module 1301 is used to acquire multiple frames of video images in a video;

[0256] The second acquisition module 1302 is configured to acquire the color temperature of any one frame of the multiple frames of video images when a preset target exists in the frame of video images;

[0257] Processing module 1303 is configured to process the frame of video image using a global color lookup table (LUT) and a first correction LUT to obtain a processed video image if the color temperature of the frame of video image is a first color temperature; and to process the frame of video image using the global LUT and the second correction LUT to obtain a processed video image if the color temperature of the frame of video image is a second color temperature. The global LUT is configured to process the video image as a whole, the first correction LUT and the second correction LUT are configured to perform color correction on an area in the video image where a preset target is located, the first color temperature is different from the second color temperature, and the first correction LUT is different from the second correction LUT.

[0258] Optionally, the preset target is a portrait, the sky, the water surface, or plants.

[0259] Optionally, the second obtaining module 1302 is configured to:

[0260] Use the AWB algorithm to estimate the color temperature of the entire frame of video image to obtain the color temperature of the frame of video image; or use the AWB algorithm to estimate the color temperature of the area where a preset target is located in the frame of video image to obtain the color temperature of the frame of video image; or use the AWB algorithm to estimate the color temperature of the neutral gray pixel points in the frame of video image to obtain the color temperature of the frame of video image.

[0261] Optionally, the second obtaining module 1302 is configured to:

[0262] The color temperature of each of the m×n regions in the frame of video image is detected by a spectral detection sensor, where the spectral detection sensor includes m×n detection elements, where m and n are both positive integers; and the color temperature of the frame of video image is determined based on the color temperature of each of the m×n regions in the frame of video image.

[0263] Optionally, the device further comprises:

[0264] The third acquisition module is configured to acquire a correction LUT corresponding to the first color temperature from the correspondence between the color temperature and the correction LUT, where the correction LUT corresponding to the first color temperature is the first correction LUT.

[0265] Optionally, the device further comprises:

[0266] a fourth acquisition module, configured to acquire a correction LUT used when processing each of k frames of video images preceding the current frame of video image in the multiple frames of video image, to obtain k correction LUTs, where k is a positive integer;

[0267] a fifth acquisition module, configured to acquire a correction LUT corresponding to the first color temperature from a correspondence between color temperatures and correction LUTs;

[0268] The calculation module is configured to perform an interpolation operation on the k correction LUTs and a correction LUT corresponding to the first color temperature to obtain a first correction LUT.

[0269] Optionally, the processing module 1303 is configured to:

[0270] The global LUT is used to process the entire frame of video image to obtain a first image; the first correction LUT is used to process the area where the preset target is located in the frame of video image to obtain a second image; the first image and the second image are fused to obtain a processed video image.

[0271] Optionally, the device further comprises:

[0272] A sixth acquisition module, used to obtain global LUT weights and correction LUT weights;

[0273] The processing module 1303 is used to:

[0274] The first image and the second image are fused according to the global LUT weight and the correction LUT weight to obtain a processed video image.

[0275] Optionally, the sixth acquisition module is used to:

[0276] A slider bar is displayed, which is used to indicate the correction LUT weight; if a sliding operation of a slider in the slider bar is detected, the weight corresponding to the position of the slider in the slider bar is determined as the correction LUT weight; and the correction LUT weight is subtracted from 1 to obtain the global LUT weight.

[0277] Optionally, the sixth acquisition module is used to:

[0278] From the correspondence between the global LUT and the global LUT weight, the global LUT weight corresponding to the global LUT is obtained; and the global LUT weight is subtracted from 1 to obtain the correction LUT weight.

[0279] Optionally, the processing module 1303 is configured to:

[0280] The global LUT is used to process the entire frame of video image to obtain a first image;

[0281] The first correction LUT is used to process the area where the preset target is located in the first image to obtain a processed video image.

[0282] Optionally, the device further comprises:

[0283] A first display module is configured to display a camera preview interface in the movie mode if a selection operation for the movie mode is detected, wherein the camera preview interface includes a LUT control for activating a LUT function;

[0284] A second display module is configured to display a LUT bar on the camera preview interface if a selection operation on the LUT control is detected, the LUT bar including a plurality of LUT icons;

[0285] The seventh acquisition module is configured to acquire the LUT corresponding to one of the multiple LUT icons as the global LUT if a selection operation on the one LUT icon among the multiple LUT icons is detected.

[0286] Optionally, the first obtaining module 1301 is configured to:

[0287] During the shooting process of the camera, multiple frames of video images in the video being shot by the camera are obtained; or,

[0288] After the camera finishes shooting, multiple frames of video images in the video shot by the camera are obtained.

[0289] In an embodiment of the present application, multiple frames of video images in a video are obtained. For any frame of video images in the multiple frames of video images, the color temperature of this frame of video image can be obtained when a preset target exists in this frame of video image. Afterwards, if the color temperature of this frame of video is the first color temperature, the global LUT and the first correction LUT are used to process this frame of video image. If the color temperature of this frame of video is the second color temperature, the global LUT and the second correction LUT are used to process this frame of video image. In this way, when the color temperature of the video image is different, different correction LUTs can be used to perform color correction on the area where the preset target is located in the video image, so that the video image processed by the global LUT and the correction LUT can not only have the filter effect corresponding to the global LUT, but also effectively avoid the color cast problem of the preset target.

[0290] It should be noted that: the video processing device provided in the above embodiment only uses the division of the above-mentioned functional modules as an example to illustrate video processing. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0291] The functional units and modules in the above embodiments may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The above integrated units may be implemented in the form of hardware or software functional units. In addition, the specific names of the functional units and modules are only for the purpose of distinguishing them from each other and are not intended to limit the scope of protection of the embodiments of this application.

[0292] The video processing device and the video processing method provided in the above embodiments belong to the same concept. The specific working process and technical effects brought about by the units and modules in the above embodiments can be found in the method embodiment part and will not be repeated here.

[0293] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, computer, server or data center to another website, computer, server or data center via a wired (such as a coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access, or a data storage device such as a server or data center that includes one or more available media integrations. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital versatile disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).

[0294] The above are optional embodiments provided for this application and are not intended to limit this application. Any modifications, equivalent replacements, improvements, etc. made within the technical scope disclosed in this application should be included in the scope of protection of this application.

Claims

1. A video processing method, characterized in that: The method comprises: Get multiple frames of video images in the video; For any one frame of video images among the multiple frames of video images, if a preset target exists in the one frame of video image, obtaining the color temperature of the one frame of video image; If the color temperature of the frame of video image is the first color temperature, use the global color lookup table LUT and the first correction LUT to process the frame of video image to obtain a processed video image; If the color temperature of the frame of video image is the second color temperature, use the global LUT and the second correction LUT to process the frame of video image to obtain a processed video image; Among them, the global LUT is used to process the video image as a whole, the first correction LUT and the second correction LUT are both used to perform color correction on the area where the preset target is located in the video image, the first color temperature is different from the second color temperature, and the first correction LUT is different from the second correction LUT.

2. The method according to claim 1, wherein The preset target is a portrait, the sky, the water surface, or a plant.

3. The method according to claim 1, wherein The obtaining of the color temperature of the frame of video image includes: Using an automatic white balance (AWB) algorithm, the color temperature of the entire frame of video image is estimated to obtain the color temperature of the frame of video image; or, Using the AWB algorithm, estimating the color temperature of the area where the preset target is located in the frame of video image to obtain the color temperature of the frame of video image; or, The AWB algorithm is used to estimate the color temperature of neutral gray pixels in the frame of video image to obtain the color temperature of the frame of video image.

4. The method according to claim 1, wherein The obtaining of the color temperature of the frame of video image includes: detecting the color temperature of each of the m×n regions in the one frame of video image by using a spectrum detection sensor, wherein the spectrum detection sensor includes m×n detection elements, and m and n are both positive integers; The color temperature of the frame of video image is determined according to the color temperature of each of the m×n regions in the frame of video image.

5. The method according to claim 1, wherein Before using the global color lookup table LUT and the first correction LUT to process the one frame of video image, the method further includes: A correction LUT corresponding to the first color temperature is obtained from the correspondence between color temperature and the correction LUT, where the correction LUT corresponding to the first color temperature is the first correction LUT.

6. The method according to claim 1, wherein Before using the global color lookup table LUT and the first correction LUT to process the one frame of video image, the method further includes: Obtaining a correction LUT used when processing each of k frames of video images preceding the one frame of video image in the multiple frames of video images, to obtain k correction LUTs, where k is a positive integer; Obtaining a correction LUT corresponding to the first color temperature from a correspondence between color temperatures and correction LUTs; An interpolation operation is performed on the k correction LUTs and the correction LUT corresponding to the first color temperature to obtain the first correction LUT.

7. The method according to claim 1, wherein The step of processing the frame of video image using the global color lookup table LUT and the first correction LUT to obtain a processed video image includes: Using the global LUT to process the entire frame of video image to obtain a first image; Using the first correction LUT to process the area where the preset target is located in the one frame of video image to obtain a second image; The first image and the second image are fused to obtain a processed video image.

8. The method according to claim 7, wherein Before acquiring multiple frames of video images in the video, the method further includes: Get global LUT weight and correction LUT weight; The fusing the first image and the second image to obtain a processed video image includes: The first image and the second image are image-fused according to the global LUT weight and the correction LUT weight to obtain a processed video image.

9. The method according to claim 8, wherein The obtaining of the global LUT weight and the correction LUT weight includes: Displaying a slider bar, the slider bar being used to indicate the correction LUT weight; If a sliding operation on the slider in the slider bar is detected, a weight corresponding to the position of the slider in the slider bar is determined as the correction LUT weight; The corrected LUT weight is subtracted from 1 to obtain the global LUT weight.

10. The method according to claim 8, wherein The obtaining of the global LUT weight and the correction LUT weight includes: Obtaining a global LUT weight corresponding to the global LUT from a correspondence between the global LUT and the global LUT weight; The global LUT weight is subtracted from 1 to obtain the corrected LUT weight.

11. The method according to claim 1, wherein The step of processing the frame of video image using the global color lookup table LUT and the first correction LUT to obtain a processed video image includes: Using the global LUT to process the entire frame of video image to obtain a first image; The first correction LUT is used to process the area where the preset target is located in the first image to obtain a processed video image.

12. The method according to any one of claims 1 to 11, characterized in that: Before acquiring multiple frames of video images in the video, the method further includes: If a selection operation for the movie mode is detected, a camera preview interface in the movie mode is displayed, wherein the camera preview interface includes a LUT control, and the LUT control is used to activate the LUT function; If a selection operation on the LUT control is detected, a LUT bar is displayed on the camera preview interface, wherein the LUT bar includes a plurality of LUT icons; If a selection operation on one of the multiple LUT icons is detected, a LUT corresponding to the one LUT icon is obtained as the global LUT.

13. The method according to any one of claims 1 to 11, characterized in that: The step of acquiring multiple frames of video images in a video includes: During the shooting process of the camera, obtaining multiple frames of video images in the video being shot by the camera; or, After the camera finishes shooting, multiple frames of video images in the video shot by the camera are obtained.

14. A video processing device, characterized in that: The device comprises: A first acquisition module is used to acquire multiple frames of video images in a video; A second acquisition module is configured to acquire, for any one frame of video images among the multiple frames of video images, a color temperature of the one frame of video image when a preset target exists in the one frame of video image; A processing module is configured to, if the color temperature of the frame of video image is a first color temperature, use a global color lookup table (LUT) and a first correction LUT to process the frame of video image to obtain a processed video image; if the color temperature of the frame of video image is a second color temperature, use a global LUT and a second correction LUT to process the frame of video image to obtain a processed video image; wherein, the global LUT is used to process the video image as a whole, the first correction LUT and the second correction LUT are both used to perform color correction on the area where the preset target is located in the video image, the first color temperature is different from the second color temperature, and the first correction LUT is different from the second correction LUT.

15. A computer device, characterized in that: The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the method according to any one of claims 1 to 13 when executed by the processor.

16. A computer-readable storage medium, characterized in that The computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the method according to any one of claims 1 to 13.

Images