Video sky region color adjusting method and device, electronic equipment and storage medium
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHUXING TECH (BEIJING) CO LTD
- Filing Date
- 2022-09-21
- Publication Date
- 2026-05-29
AI Technical Summary
Existing sky segmentation models are prone to missegmentation when dealing with areas such as snow and water that are similar in color to the sky. This can lead to abnormal inter-frame rendering and jitter, affecting the aesthetics and viewing experience of the video.
By segmenting video frames, historical information from sky segmentation is used to adjust for misidentification. An initial binary image and a threshold are used to identify the sky region, and the processing method for missegmented frames is adjusted to avoid rendering anomalies.
Without changing the model parameters, the rendering anomalies and jitter issues between video frames were resolved, improving the aesthetics and viewing experience of the video.
Smart Images
Figure CN117788506B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of video processing technology, specifically to a method, apparatus, electronic device, and storage medium for color correction of the sky area in a video. Background Technology
[0002] Given users' increasing demands for accurate sky color reproduction in videos, sky color correction algorithms can adjust the color of the sky area to meet user needs. However, sky color correction relies on the ability to segment a complete sky area without significant missegmentation. In scenes with blurred sky boundaries, or when there are areas in the image that closely resemble the sky, such as snowfields or water reflecting the sky, sky segmentation becomes extremely challenging, especially in long videos, due to the similarity in sky color to these areas.
[0003] Currently, mainstream sky segmentation models are often only applicable to general scenarios. They can perform well for specific scenarios after training with business data, but they will show obvious segmentation errors for large areas of similar sky color, such as snow or water. This will lead to rendering anomalies and jitter between video frames after rendering, which will affect the aesthetics of the video and the viewing experience of the viewer. Summary of the Invention
[0004] To address the aforementioned problems in the prior art, this application provides a video sky region color correction method, apparatus, electronic device, and storage medium. It can adjust the segmentation of subsequent images based on historical sky segmentation information, and handle missegmentation without changing model parameters. This solves the problems of abnormal rendering and jitter in the model output between video frames, thereby improving the video's aesthetics and the viewer's viewing experience.
[0005] In a first aspect, embodiments of this application provide a method for color correction of the sky region in a video, the method comprising:
[0006] The video data is divided into frames to obtain N video frame images;
[0007] Sky region identification is performed on the first video frame image out of N video frames to obtain an initial binary image;
[0008] The first threshold is determined based on the size information of the first video frame image;
[0009] Based on the initial binary image and the first threshold, N-1 sky region recognition processes are performed on N-1 video frame images to obtain N-1 final binary images. The N-1 video frame images are the N-1 images remaining after removing the first video frame image from the N video frame images.
[0010] Based on the initial binary image and N-1 final binary images, the sky regions of N video frames are color-corrected to obtain the color-corrected video data.
[0011] Secondly, embodiments of this application provide a video sky area color correction device, comprising:
[0012] The frame segmentation module is used to process video data into frames to obtain N video frame images;
[0013] The processing module is used to perform sky region recognition on the first video frame image out of N video frame images to obtain an initial binary image. Based on the size information of the first video frame image, a first threshold is determined. Based on the initial binary image and the first threshold, the sky region recognition processing is performed on N-1 video frame images to obtain N-1 final binary images. The N-1 video frame images are the N-1 images remaining after removing the first video frame image from the N video frame images.
[0014] The color grading module is used to perform color grading on N video frame images, dividing them into sky regions, based on the initial binary image and N-1 final binary images, to obtain the color-graded video data.
[0015] Thirdly, embodiments of this application provide an electronic device, including: a processor connected to a memory for storing a computer program, and the processor for executing the computer program stored in the memory to cause the electronic device to perform the method as described in the first aspect.
[0016] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that causes a computer to perform the method as described in the first aspect.
[0017] Fifthly, embodiments of this application provide a computer program product, the computer program product including a non-transitory computer-readable storage medium storing a computer program, and a computer operable to perform the method as described in the first aspect.
[0018] Implementing the embodiments of this application has the following beneficial effects:
[0019] In this embodiment, after obtaining the sky segmentation result for each frame in the video, the misidentification situation of each frame is determined based on the historical information of sky segmentation, and then the misidentified frames are adjusted accordingly. Thus, without changing the model parameters, the missegmentation phenomenon is processed, solving the problems of abnormal rendering and jitter in the model output between video frames, improving the video's aesthetics and the viewer's viewing experience. Attached Figure Description
[0020] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0021] Figure 1 A schematic diagram of the hardware structure of a video sky area color correction device provided for an embodiment of this application;
[0022] Figure 2 A flowchart illustrating a video sky area color correction method provided for an embodiment of this application;
[0023] Figure 3 A schematic diagram of an initial binary image provided for an embodiment of this application;
[0024] Figure 4 A functional module block diagram of a video sky area color correction device provided for embodiments of this application;
[0025] Figure 5 This is a schematic diagram of the structure of an electronic device provided for an embodiment of this application. Detailed Implementation
[0026] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, not all of them. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0027] The terms "first," "second," "third," and "fourth," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.
[0028] In this document, the term "implementation" means that a specific feature, result, or characteristic described in connection with an implementation may be included in at least one implementation of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same implementation, nor is it a separate or alternative implementation mutually exclusive with other implementations. It will be explicitly and implicitly understood by those skilled in the art that the implementations described herein can be combined with other implementations.
[0029] See Figure 1 , Figure 1 This is a schematic diagram of the hardware structure of a video sky area color correction device provided in an embodiment of this application. The video sky area color correction device 100 includes at least one processor 101, a communication line 102, a memory 103, and at least one communication interface 104.
[0030] In this embodiment, the processor 101 may be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of the program of the present application.
[0031] Communication line 102 may include a path for transmitting information between the aforementioned components.
[0032] The communication interface 104 can be any transceiver-like device (such as an antenna) used to communicate with other devices or communication networks, such as Ethernet, RAN, wireless local area networks (WLAN), etc.
[0033] The memory 103 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.
[0034] In this embodiment, the memory 103 can exist independently and be connected to the processor 101 via the communication line 102. Alternatively, the memory 103 can be integrated with the processor 101. The memory 103 provided in this embodiment is typically non-volatile. The memory 103 stores computer execution instructions for implementing the scheme of this application, and its execution is controlled by the processor 101. The processor 101 executes the computer execution instructions stored in the memory 103 to implement the method provided in the following embodiments of this application.
[0035] In an optional implementation, the computer execution instructions may also be referred to as application code, and this application does not specifically limit this terminology.
[0036] In an optional implementation, processor 101 may include one or more CPUs, for example... Figure 1 CPU0 and CPU1 in the CPU.
[0037] In an optional implementation, the video sky area color correction device 100 may include multiple processors, such as... Figure 1 Processors 101 and 107 are shown in the diagram. Each of these processors may be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. A processor here may refer to one or more devices, circuits, and / or processing cores used to process data (e.g., computer program instructions).
[0038] In optional embodiments, if the video sky area color correction device 100 is a server, for example, it can be a standalone server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Then the video sky area color correction device 100 may further include an output device 105 and an input device 106. The output device 105 communicates with the processor 101 and can display information in various ways. For example, the output device 105 can be a liquid crystal display (LCD), a light emitting diode (LED) display device, a cathode ray tube (CRT) display device, or a projector, etc. The input device 106 communicates with the processor 101 and can receive user input in various ways. For example, the input device 106 can be a mouse, keyboard, touchscreen device, or sensor device, etc.
[0039] The aforementioned video sky area color correction device 100 can be a general-purpose device or a special-purpose device. This application does not limit the type of video sky area color correction device 100.
[0040] The following will explain the color correction method for the video sky area disclosed in this application:
[0041] See Figure 2 , Figure 2 This is a flowchart illustrating a method for color grading the sky area in a video, provided as an embodiment of this application. The method includes the following steps:
[0042] 201: Perform frame segmentation on the video data to obtain N video frame images.
[0043] In this embodiment, N is an integer greater than 0. Specifically, the N video frame images obtained from framing can be sorted according to their temporal order in the video data.
[0044] 202: Perform sky region identification on the first video frame image out of N video frame images to obtain an initial binary image.
[0045] In this embodiment, any segmentation method or model capable of sky segmentation within the art can be used to identify the sky region in the first video frame image, obtaining the sky segmentation result, i.e., the initial binary image, denoted as mask_frame1. In this embodiment, the sky segmentation result is represented in the form of a binary image mask. Specifically, as follows... Figure 3 As shown, the initial binary image mask is actually a matrix of values from 0 to 1, where each value corresponds to a pixel in the video frame image, and 0 represents a non-sky area, which appears black in the image, and 1 represents a sky area, which appears white in the image.
[0046] 203: Determine the first threshold based on the size information of the first video frame image.
[0047] In this embodiment, before step 203, the recognition effect of the initial binary image mask_frame1 can be determined first. Specifically, any method or model in the art that can realize sky segmentation effect recognition can be used to identify the recognition effect of the initial binary image mask_frame1, and this application does not limit this.
[0048] In this embodiment, when the recognition effect meets preset recognition conditions, such as the accuracy rate being greater than a preset threshold, the mean vertical coordinate of the sky region in the initial binary image can be calculated and denoted as dynamic_threshold. The height of the first video frame image, or the height of the video data frame, is obtained and denoted as height. Then, a relaxation variable is determined based on the height of the first video frame image, and the first threshold is determined based on the mean vertical coordinate of the sky region in the initial binary image, dynamic_threshold, and the relaxation variable height.
[0049] Specifically, slack variables can be represented by formula ①:
[0050] margin=k*height………①
[0051] Where margin is a slack variable, k is a hyperparameter, and height is the height of the first video frame.
[0052] In an optional implementation, the value of k can be 0.1.
[0053] Furthermore, the first threshold can be expressed by formula ②:
[0054] L1=dynamic_threshold–margin…………②
[0055] Where L1 is the first threshold, dynamic_threshold is the mean of the ordinate of the sky region in the initial binary image, and margin is the slack variable.
[0056] Furthermore, in this embodiment, when the recognition effect does not meet the preset recognition conditions, half the height of the first video frame image can be directly used as the first threshold. That is, in this case, the first threshold can be expressed by formula ③:
[0057] L'1 = height / 2………③
[0058] Where height is the height of the first video frame image.
[0059] 204: Based on the initial binary image and the first threshold, perform N-1 sky region recognition processes on N-1 video frame images to obtain N-1 final binary images.
[0060] In this embodiment, N-1 video frame images are the N-1 images remaining after removing the first video frame image from the N video frame images. For example, in the i-th sky region recognition process, the i-th video frame image A can first be processed... i Sky region identification is performed to obtain binary image B. i Where i is an integer greater than 1 and less than N. Specifically, for the i-th video frame image A... i The method for sky region identification is similar to the method for sky region identification in step 202 for the first video frame image among N video frame images, and will not be described again here.
[0061] Then, the binary image B can be calculated. i With historical binary graph C i The crossover ratio between them, where, when i = 2, the historical binary graph C i This is the initial binary image. Simultaneously, calculate binary image B. i Mean ordinate of the mid-sky region.
[0062] Then, if the intersection-union ratio is greater than or equal to the preset second threshold and the mean of the ordinate is less than or equal to the first threshold, it indicates that the sky segmentation of the frame image is correct and there is no missegmentation. In this case, the i-th video frame image A can be directly determined. i The final binary image D i For binary image B i .
[0063] At this point, the final binary image D can be obtained. i The historical binary image C used in the (i+1)th sky region identification process i+1The process involves performing the (i+1)th sky region identification process, continuing until the N-1th sky region identification process is performed, resulting in N-1 final binary images. In short, when the segmentation is correct, the historical binary image is updated to reflect the segmentation result of the current frame, and adjustments are made to the sky region identification process for the next frame.
[0064] Meanwhile, in this embodiment, when the intersection-union ratio is less than the second threshold or the mean of the ordinate is greater than the first threshold, it indicates that the sky segmentation of the frame image is incorrect, and there is a case of missegmentation. In this case, the binary image B is... i Setting it to zero yields the misidentified binary image E. i And determine the final binary image D of the i-th video frame image Ai. i For misidentified binary image E i Therefore, the final binary image D of this image is... i A matrix consisting entirely of zeros indicates that the image frame does not contain any sky areas, and therefore no color correction or rendering is performed on it, thus preventing rendering anomalies.
[0065] At this point, the historical binary graph C can be... i The historical binary image C used in the (i+1)th sky region identification process i+1 The process involves performing the (i+1)th sky region identification process, up to the N-1th time, to obtain N-1 final binary images. In short, when missegmentation occurs, the historical binary images are not updated, and the next frame's sky region identification process is performed directly.
[0066] 205: Based on the initial binary image and N-1 final binary images, perform color correction on N video frame images by dividing the sky region to obtain the color-corrected video data.
[0067] In this embodiment, after determining the binary image of each of the N video frames, the area to be color-corrected can be identified in each frame based on the sky region (i.e., the region with a value of 1) in the corresponding binary image. Finally, the color-corrected N frames are merged to obtain the color-corrected video data.
[0068] The following will illustrate the video sky area color correction method in this application with a specific example:
[0069] First, the sky is segmented in each frame of the video data using a sky segmentation model, resulting in the segmentation results: [mask1, mask2, ..., maskN]. The segmentation result mask1 of the first frame is stored as history, i.e., history = mask1, for use in intersection-over-union (IoU) calculations for subsequent frames.
[0070] At this point, the segmentation effect of the segmentation result mask1 of the first frame can be judged. When its segmentation effect meets the conditions, a dynamic threshold strategy can be adopted. Specifically, the mean y-coordinate of the sky region marked as 1 in the segmentation result mask1 of the first frame can be calculated and denoted as y. d Simultaneously, the height of the first frame image is obtained and denoted as h; and the slack variable m = 0.1 * h is set to calculate the first threshold p1 = y. d –m.
[0071] When the segmentation effect does not meet the conditions, the first threshold is directly set to half the height of the first frame image, that is, p1 = h / 2.
[0072] From frame 1 to frame N, the process is as follows:
[0073] In the first frame, since there is no first frame, the final segmentation result mask'1 of the first frame is directly set to mask1, that is, mask'1 = mask1.
[0074] In the second frame, firstly, calculate the intersection-over-union ratio (IoU2) between the segmentation result mask2 and history, and the mean y-coordinate of the sky region in mask2. 2 d .
[0075] When iou2 is less than 0.5 or y 2 d If the value is greater than p1, then the segmentation result mask2 of the second frame is determined to be a missegmentation. In this case, the final segmentation result mask'2 of the second frame can be set to all 0s, meaning no sky appears, and no sky color adjustment is performed for the current frame. At the same time, the history is not updated, and processing of the third frame is started directly.
[0076] When iou2 is greater than or equal to 0.5 or y 2 d If p1 is less than or equal to p1, then the segmentation result mask2 of the second frame is considered correct. At this point, the final segmentation result mask'2 of the second frame can be set to mask2, i.e., mask'2 = mask2. Simultaneously, the history is updated to the final segmentation result mask'2 of the second frame, i.e., history = mask'2, before proceeding to process the third frame.
[0077] Finally, for frames 3, 4, ..., N, the above processing of frame 2 is repeated, and the final segmentation result of each frame is: [mask'1, mask'2, ..., mask'N].
[0078] In summary, the video sky region color correction method provided by this invention, after obtaining the sky segmentation result for each frame in the video, determines the misidentification situation of each frame based on the historical information of sky segmentation, and then performs corresponding adjustment processing on the misidentified frames. Thus, without changing the model parameters, it handles the missegmentation phenomenon, solves the problems of abnormal rendering and jitter between video frames in the model output, and improves the aesthetics of the video and the viewing experience of the viewer.
[0079] See Figure 4 , Figure 4 This is a block diagram illustrating the functional modules of a video sky area color correction device provided for an embodiment of this application. (For example...) Figure 4 As shown, the video sky area color correction device 400 includes:
[0080] The frame segmentation module 401 is used to segment video data into frames to obtain N video frame images.
[0081] The processing module 402 is used to perform sky region recognition on the first video frame image in N video frame images to obtain an initial binary image, determine a first threshold based on the size information of the first video frame image, and perform N-1 sky region recognition processing on N-1 video frame images based on the initial binary image and the first threshold to obtain N-1 final binary images, wherein the N-1 video frame images are the N-1 images remaining after removing the first video frame image from the N video frame images;
[0082] The color grading module 403 is used to perform color grading on N video frame images by sky region based on the initial binary image and N-1 final binary images, so as to obtain the color-graded video data.
[0083] In an embodiment of the present invention, in performing N-1 sky region recognition processes on N-1 video frame images based on an initial binary image to obtain N-1 final binary images, the processing module 402 is specifically used for:
[0084] In the i-th sky region identification process, for the i-th video frame image A i Sky region identification is performed to obtain binary image B. i , where i is an integer greater than 1 and less than N;
[0085] Calculate binary image B i With historical binary graph C i The crossover ratio between them, where, when i = 2, the historical binary graph C i This is the initial binary image;
[0086] Calculate binary image B i Mean ordinate of the mid-sky region;
[0087] When the intersection-union ratio is greater than or equal to a preset second threshold and the mean of the ordinate is less than or equal to a first threshold, the i-th video frame image A is determined. i The final binary image D i For binary image B i ;
[0088] The final binary image D i The historical binary image C used in the (i+1)th sky region identification process i+1 Perform the (i+1)th sky region identification process until the (N-1)th sky region identification process is performed, resulting in N-1 final binary images.
[0089] In an embodiment of the present invention, in obtaining N-1 final binary images by performing N-1 sky region recognition processes on N-1 video frame images based on the initial binary image, the processing module 402 is further configured to:
[0090] When the intersection-union ratio is less than the second threshold or the mean of the ordinate is greater than the first threshold, the binary image B will be... i Setting it to zero yields the misidentified binary image E. i ;
[0091] Determine the final binary image D of the i-th video frame image Ai. i For misidentified binary image E i ;
[0092] The historical binary graph C i The historical binary image C used in the (i+1)th sky region identification process i+1 Perform the (i+1)th sky region identification process until the (N-1)th sky region identification process is performed, resulting in N-1 final binary images.
[0093] In an embodiment of the present invention, after performing sky region identification on the first video frame image among N video frame images to obtain an initial binary image, the processing module 402 is further configured to:
[0094] Determine the recognition performance of the initial binary image;
[0095] When the recognition effect meets the preset recognition conditions, a first threshold is determined based on the size information of the first video frame image, including:
[0096] Calculate the mean ordinate of the sky region in the initial binary image;
[0097] Obtain the height of the first video frame image, and determine the slack variable based on the height of the first video frame image;
[0098] The first threshold is determined based on the mean of the ordinate of the sky region and the slack variable in the initial binary image.
[0099] In an embodiment of the present invention, the slack variable can be represented by formula ④:
[0100] margin=k*height………④
[0101] Where margin is a slack variable, k is a hyperparameter, and height is the height of the first video frame.
[0102] In an embodiment of the present invention, the first threshold can be expressed by formula ⑤:
[0103] L1=dynamic_threshold–margin…………⑤
[0104] Where L1 is the first threshold, dynamic_threshold is the mean of the ordinate of the sky region in the initial binary image, and margin is the slack variable.
[0105] See Figure 5 , Figure 5 This is a schematic diagram of the structure of an electronic device provided for an embodiment of this application. For example... Figure 5 As shown, the electronic device 500 includes a transceiver 501, a processor 502, and a memory 503. These are connected via a bus 504. The memory 503 stores computer programs and data, and can transfer data stored in the memory 503 to the processor 502.
[0106] Processor 502 is used to read the computer program in memory 503 and perform the following operations:
[0107] The video data is divided into frames to obtain N video frame images;
[0108] Sky region identification is performed on the first video frame image out of N video frames to obtain an initial binary image;
[0109] The first threshold is determined based on the size information of the first video frame image;
[0110] Based on the initial binary image and the first threshold, N-1 sky region recognition processes are performed on N-1 video frame images to obtain N-1 final binary images. The N-1 video frame images are the N-1 images remaining after removing the first video frame image from the N video frame images.
[0111] Based on the initial binary image and N-1 final binary images, the sky regions of N video frames are color-corrected to obtain the color-corrected video data.
[0112] In an embodiment of the present invention, in performing N-1 sky region identification processes on N-1 video frame images based on an initial binary image to obtain N-1 final binary images, the processor 502 is specifically configured to perform the following operations:
[0113] In the i-th sky region identification process, for the i-th video frame image A i Sky region identification is performed to obtain binary image B. i , where i is an integer greater than 1 and less than N;
[0114] Calculate binary image B i With historical binary graph C i The crossover ratio between them, where, when i = 2, the historical binary graph C i This is the initial binary image;
[0115] Calculate binary image B i Mean ordinate of the mid-sky region;
[0116] When the intersection-union ratio is greater than or equal to a preset second threshold and the mean of the ordinate is less than or equal to a first threshold, the i-th video frame image A is determined. i The final binary image D i For binary image B i ;
[0117] The final binary image D i The historical binary image C used in the (i+1)th sky region identification process i+1 Perform the (i+1)th sky region identification process until the (N-1)th sky region identification process is performed, resulting in N-1 final binary images.
[0118] In an embodiment of the present invention, in order to perform N-1 sky region identification processes on N-1 video frame images based on the initial binary image to obtain N-1 final binary images, the processor 502 is further configured to perform the following operations:
[0119] When the intersection-union ratio is less than the second threshold or the mean of the ordinate is greater than the first threshold, the binary image B will be... i Setting it to zero yields the misidentified binary image E. i ;
[0120] Determine the final binary image D of the i-th video frame image Ai. i For misidentified binary image E i ;
[0121] The historical binary graph C i The historical binary image C used in the (i+1)th sky region identification process i+1Perform the (i+1)th sky region identification process until the (N-1)th sky region identification process is performed, resulting in N-1 final binary images.
[0122] In an embodiment of the present invention, after performing sky region identification on the first video frame image out of N video frame images to obtain an initial binary image, the processor 502 is further configured to perform the following operations:
[0123] Determine the recognition performance of the initial binary image;
[0124] When the recognition effect meets the preset recognition conditions, a first threshold is determined based on the size information of the first video frame image, including:
[0125] Calculate the mean ordinate of the sky region in the initial binary image;
[0126] Obtain the height of the first video frame image, and determine the slack variable based on the height of the first video frame image;
[0127] The first threshold is determined based on the mean of the ordinate of the sky region and the slack variable in the initial binary image.
[0128] In an embodiment of the present invention, the slack variable can be represented by formula ⑥:
[0129] margin = k * height ………⑥
[0130] Where margin is a slack variable, k is a hyperparameter, and height is the height of the first video frame.
[0131] In an embodiment of the present invention, the first threshold can be represented by formula ⑦:
[0132] L1=dynamic_threshold–margin…………⑦
[0133] Where L1 is the first threshold, dynamic_threshold is the mean of the ordinate of the sky region in the initial binary image, and margin is the slack variable.
[0134] It should be understood that the video sky area color grading device in this application may include smartphones (such as Android phones, iOS phones, Windows Phones, etc.), tablet computers, PDAs, laptops, mobile internet devices (MIDs), robots, or wearable devices, etc. The above-mentioned video sky area color grading devices are merely examples and not exhaustive, and include, but are not limited to, the aforementioned video sky area color grading devices. In practical applications, the above-mentioned video sky area color grading device may also include: intelligent vehicle terminals, computer equipment, etc.
[0135] Through the above description of the embodiments, those skilled in the art can clearly understand that the present invention can be implemented by means of software combined with a hardware platform. Based on this understanding, all or part of the technical solution of the present invention that contributes to the background art can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of the present invention.
[0136] Therefore, embodiments of this application also provide a computer-readable storage medium storing a computer program that is executed by a processor to implement some or all of the steps of any of the video sky area color correction methods described in the above method embodiments. For example, the storage medium may include a hard disk, floppy disk, optical disk, magnetic tape, magnetic disk, USB flash drive, flash memory, etc.
[0137] This application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps of any of the video sky area color grading methods described in the above method embodiments.
[0138] It should be noted that, for the sake of simplicity, the aforementioned methods are described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are optional, and the actions and modules involved are not necessarily essential to this application.
[0139] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0140] In the several embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical or other forms.
[0141] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.
[0142] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software program module.
[0143] If the integrated unit is implemented as a software program module and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.
[0144] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage device, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.
[0145] The embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The above description of the embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.
Claims
1. A method for color correction of the sky area in a video, characterized in that, The method includes: The video data is divided into frames to obtain N video frame images; Sky region identification is performed on the first video frame image among the N video frame images to obtain an initial binary image; Based on the size information of the first video frame image, a first threshold is determined; Based on the initial binary image and the first threshold, N-1 sky region recognition processes are performed on N-1 video frame images to obtain N-1 final binary images, wherein the N-1 video frame images are the N-1 images remaining after removing the first video frame image from the N video frame images. Based on the initial binary image and the N-1 final binary images, the N video frame images are color-corrected by dividing the sky region to obtain the color-corrected video data. The step of performing N-1 sky region recognition processes on N-1 video frame images based on the initial binary image and the first threshold to obtain N-1 final binary images includes: In the i-th sky region identification process, for the i-th video frame image A i Sky region identification is performed to obtain binary image B. i , where i is an integer greater than 1 and less than N; Calculate the binary image B i With historical binary graph C i The crossover ratio between them, where, when i=2, the historical binary graph C i This is the initial binary image; Calculate the binary image B i Mean ordinate of the mid-sky region; When the intersection-union ratio is less than the second threshold or the mean value of the ordinate is greater than the first threshold, the binary image B is... i Setting it to zero yields the misidentified binary image E. i ; Determine the i-th video frame image A i The final binary image D i For the misidentified binary image E i ; The historical binary image C i The historical binary image C used in the (i+1)th sky region identification process i+1 The i+1th sky region identification process is performed until the N-1th sky region identification process is performed, resulting in the N-1 final binary images.
2. The method according to claim 1, characterized in that, The step of performing N-1 sky region recognition processes on N-1 video frame images based on the initial binary image and the first threshold to obtain N-1 final binary images further includes: When the intersection-union ratio is greater than or equal to a preset second threshold and the mean of the ordinate is less than or equal to the first threshold, the i-th video frame image A is determined. i The final binary image D i For the binary image B i ; The final binary image D i The historical binary image C used in the (i+1)th sky region identification process i+1 The i+1th sky region identification process is performed until the N-1th sky region identification process is performed, resulting in the N-1 final binary images.
3. The method according to claim 1, characterized in that, After performing sky region identification on the first video frame image among the N video frame images to obtain an initial binary image, the method further includes: Determine the recognition performance of the initial binary image; When the recognition effect meets the preset recognition conditions, determining the first threshold based on the size information of the first video frame image includes: Calculate the mean ordinate of the sky region in the initial binary image; Obtain the height of the first video frame image, and determine the relaxation variable based on the height of the first video frame image; The first threshold is determined based on the mean ordinate of the sky region in the initial binary image and the slack variable.
4. The method according to claim 3, characterized in that, The slack variable is determined by the following formula: Where margin is a slack variable, k is a hyperparameter, and height is the height of the first video frame image.
5. The method according to claim 3 or 4, characterized in that, The first threshold is determined by the following formula: L1 = dynamic_threshold - margin Where L1 is the first threshold, dynamic_threshold is the mean of the ordinate of the sky region in the initial binary image, and margin is the slack variable.
6. A video sky area color correction device, the device being used to perform the method as described in any one of claims 1-5, characterized in that, The device includes: The frame segmentation module is used to process video data into frames to obtain N video frame images; The processing module is used to perform sky region recognition on the first video frame image among the N video frame images to obtain an initial binary image, determine a first threshold based on the size information of the first video frame image, and perform N-1 sky region recognition processing on the N-1 video frame images based on the initial binary image and the first threshold to obtain N-1 final binary images, wherein the N-1 video frame images are the N-1 images remaining after removing the first video frame image from the N video frame images; The color correction module is used to perform color correction on the N video frame images by dividing the sky region based on the initial binary image and the N-1 final binary images, so as to obtain the color-corrected video data.
7. An electronic device, characterized in that, The method includes a processor, a memory, a communication interface, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the processor, and the one or more programs include instructions for performing the steps of the method according to any one of claims 1-5.
8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that is executed by a processor to implement the method as described in any one of claims 1-5.