Black and white video coloring method and device and related medium

Abstract

The invention discloses a black and white video coloring method and device and a related medium, and the method comprises the steps: carrying out the threshold detection and judgment of a difference feature sequence obtained through calculation, so as to determine a key frame set of each segment; performing cue word construction by using the key frame set to obtain a cue word set, and then performing condition image generation to obtain a coloring key frame set; and performing intra-fragment reference relation modeling according to the coloring key frame set, establishing color propagation mapping, performing frame-by-frame application, completing coloring of non-key frames in fragments, and performing synthesis according to a time sequence to obtain a colored video. According to the method, the calculated coloring key frame set is used for carrying out intra-fragment reference relation modeling, establishing color propagation mapping, then carrying out frame-by-frame application, completing coloring of the non-key frames in the fragments and synthesizing according to the time sequence to obtain the colored video, so that the coloring of the same object between different frames can be kept consistent when the black-and-white video is colored.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a black and white video coloring method and device and related media. BACKGROUND

[0002] In the prior art, for coloring processing of black and white video, a convolutional neural network or a generative adversarial network is usually used to colorize the video frame by frame. Such a method mainly performs color inference based on the semantic and texture information of a single frame, and lacks effective modeling and constraint on the cross-frame correlation in a video sequence. For the case where there is scene transition, target motion or illumination change, the model is difficult to maintain the stability of color allocation in the time dimension, and is prone to cause color drift and time sequence flicker of the same object between adjacent frames, thereby destroying the overall visual effect and consistency and causing inconsistent coloring of the same object between different frames. SUMMARY

[0003] Embodiments of the present application provide a black and white video coloring method, device and related media, aiming to solve the technical problem of inconsistent coloring of the same object between different frames in the coloring process of a black and white video in the prior art.

[0004] In a first aspect, embodiments of the present application provide a black and white video coloring method, comprising: extracting the luminance component of the color space and calculating the difference between adjacent frames for each input black and white video frame to obtain a difference feature sequence for describing the change between frames and a corresponding frame identification sequence; performing threshold detection on the difference feature sequence and dividing the frame identification sequence into multiple segments based on the determination result to determine the key frame set of each segment; performing prompt word construction using the key frame set to obtain a prompt word set for driving conditional coloring; performing conditional image generation using the prompt word set to obtain a coloring key frame set corresponding to the key frame set; modeling the reference relationship within a segment according to the coloring key frame set, combining the frame sequence position of each segment, and establishing a color propagation mapping; performing frame-by-frame application based on the color propagation mapping, completing the coloring of non-key frames within the segment and synthesizing in time sequence to obtain a colored video.

[0005] In a second aspect, embodiments of the present application provide a black and white video coloring device, comprising: a picture extraction unit configured to extract the luminance component of the color space and calculate the difference between adjacent frames for each input black and white video frame to obtain a difference feature sequence for describing the change between frames and a corresponding frame identification sequence; A threshold determination unit is used to perform threshold detection and determination on the differential feature sequence, and divide the frame identifier sequence into multiple segments based on the determination result to determine the key frame set of each segment; A prompt construction unit is used to construct prompt words using the keyframe set to obtain a prompt word set for driving conditional coloring. An image generation unit is used to generate a conditional image using the set of prompt words to obtain a set of colorized keyframes corresponding to the set of keyframes. The color mapping unit is used to model intra-segment reference relationships based on the set of coloring keyframes, and to establish a color propagation mapping by combining the frame sequence positions of each segment. The video output unit is used to perform frame-by-frame application based on the color propagation mapping, complete the colorization of non-key frames within the segment, and synthesize them in chronological order to obtain a colorized video.

[0006] Thirdly, embodiments of the present invention provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the black-and-white video colorization method of the first aspect.

[0007] Fourthly, embodiments of the present invention provide a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, it implements the black-and-white video colorization method of the first aspect.

[0008] This invention provides a method for colorizing black and white video, including extracting the luminance component of the color space and calculating the difference between adjacent frames from the input black and white video frames to obtain a difference feature sequence describing inter-frame changes and a corresponding frame identifier sequence; performing threshold detection on the difference feature sequence and dividing the frame identifier sequence into multiple segments based on the determination result to determine the keyframe set for each segment; constructing cue words using the keyframe set to obtain a cue word set for driving conditional colorization; generating conditional images using the cue word set to obtain a colorization keyframe set corresponding to the keyframe set; modeling intra-segment reference relationships based on the colorization keyframe set, and establishing a color propagation mapping by combining the frame sequence position of each segment; applying the color propagation mapping frame by frame to complete the colorization of non-key frames within the segment and synthesizing them in chronological order to obtain a colorized video. This invention utilizes the calculated colorization keyframe set to model intra-segment reference relationships, establishes a color propagation mapping, and then applies it frame by frame to complete the colorization of non-key frames within the segment and synthesize them in chronological order to obtain a colorized video. This ensures that the colorization of the same object remains consistent across different frames during black and white video colorization.

[0009] This invention also provides a black-and-white video colorization device, a computer device, and a storage medium, which have the same beneficial effects as described above. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 A flowchart illustrating a black-and-white video colorization method provided in an embodiment of the present invention; Figure 2 This is a schematic block diagram of a black-and-white video colorization device provided in an embodiment of the present invention. Detailed Implementation

[0012] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0013] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0014] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0015] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0016] Please see below. Figure 1 , Figure 1 The flowchart of a black and white video colorization method provided in an embodiment of the present invention specifically includes steps S101 to S106.

[0017] S101. Extract the luminance component of the color space and calculate the difference between adjacent frames for the input black and white video frames respectively to obtain the difference feature sequence and the corresponding frame identifier sequence used to describe the inter-frame changes. S102. Threshold detection and determination are performed on the differential feature sequence, and the frame identifier sequence is divided into multiple segments based on the determination result to determine the key frame set of each segment; S103. Construct cue words using the keyframe set to obtain a cue word set for driving conditional coloring; S104. Use the set of prompt words to generate a conditional image to obtain a set of colorized keyframes corresponding to the set of keyframes. S105. Based on the set of coloring keyframes, perform intra-fragment reference relationship modeling, and establish a color propagation mapping by combining the frame sequence positions of each fragment; S106. Apply the color propagation mapping frame by frame to colorize the non-key frames within the segment and synthesize them in chronological order to obtain the colored video.

[0018] In step S101, the input video is read in chronological order, and a frame identifier corresponding to time is established frame by frame. Color space conversion is performed on each frame and the luminance component is extracted. The difference between the luminance components of adjacent frames is calculated to obtain the difference feature sequence, and the corresponding frame identifier is recorded synchronously to obtain a frame identifier sequence that corresponds one-to-one with the difference features, which is used to characterize the intensity and positional relationship of the changes between frames.

[0019] In one embodiment, step S101 includes: The input black and white video is processed in chronological order to establish a frame identifier that corresponds one-to-one with time. The frame identifier is converted frame by frame into a representation of hue-saturation-luminance, and luminance components are extracted from each frame to obtain a luminance component sequence. By performing pixel-by-pixel difference calculation between the current frame and the previous frame in the brightness component sequence, the difference image corresponding to the current frame is obtained; Perform full-pixel statistics on the difference image to obtain a difference metric value used to measure the intensity of change between the current frame and the previous frame; The difference metric value is paired and stored with the corresponding current frame identifier to obtain a difference feature pair; The difference features are accumulated in frame identifier order to generate a difference feature sequence and a corresponding frame identifier sequence.

[0020] In this embodiment, the black and white video is read frame by frame in chronological order, and a frame identifier corresponding to each time is established for each frame, resulting in a time-incrementing sequence of frame identifiers. For each frame in the sequence, a color space conversion is performed using the hue-saturation-luminance three-component representation, uniformly converting it to HSV format, and extracting the luminance (V) channel image from it, thus obtaining the luminance component sequence. The above conversion and extraction ensure that frames at different times are compared within the same luminance representation domain, providing consistent input for subsequent difference calculations. In the luminance component sequence, pixel-by-pixel difference calculation is performed on adjacent frames. The luminance channel image of the current frame is selected and compared with the luminance channel image of the previous frame. The difference is calculated according to the pixel position and the absolute value is taken to obtain the difference image corresponding to the current frame. For the first frame in the sequence, its difference image can be regarded as a zero image or not included in the statistics to avoid abnormal measurements caused by the lack of a preceding reference frame. The difference image completely preserves the distribution of changes in luminance of the current frame relative to the previous frame, which is used to quantify the impact of motion, scene subtle changes, and texture changes on luminance.

[0021] Furthermore, full-pixel statistics are performed on the differential image corresponding to the current frame, and the mean of all pixel differences is calculated to obtain a difference metric, the mean_v, used to measure the intensity of change between the current frame and the previous frame. Simultaneously, the current frame identifier ID corresponding to this mean difference is recorded, and the two are stored in pairs to obtain difference feature pairs. Through the above statistics, pixel-level changes can be aggregated into frame-level intensity indicators, reducing the interference of local noise on the judgment while retaining change information that can be used for subsequent segment recognition and keyframe selection. Finally, the difference feature pairs obtained over time are accumulated according to the frame identifier order and appended sequentially to the feature storage structure to form a complete difference feature sequence; the corresponding frame identifiers are accumulated in the same order to obtain a frame identifier sequence that corresponds one-to-one with the difference features.

[0022] In step S102, threshold detection is performed based on the difference feature sequence to identify boundary points with significant changes, and the frame identifier sequence is divided into multiple segments accordingly. Within each segment, key frames that represent the segment content are selected using the stable interval of the difference features in the time dimension as a reference, resulting in a set of key frames. The joint determination of difference measurement and frame identifier is used as the reference for segment division to ensure that the key frames originate from within the segment and are representative.

[0023] In one embodiment, step S102 includes: Based on the difference feature sequence, the difference in luminance components between the current frame and the previous frame is statistically analyzed for all pixels to obtain the average luminance value and the current frame identifier corresponding to the current frame. The average brightness value is compared with a preset threshold, and combined with the current frame identifier for joint determination to obtain the frame identifier sequence of the current segment; The frame identifier sequences are paired in chronological order to obtain discrete feature sequences for keyframe filtering; The total number of points in the discrete feature sequence is counted, and half of the total number of points is set as the window size. Starting from the first point of the sequence, continuous point sets are extracted sequentially according to the window size to obtain multiple candidate window segments. Calculate the mean variance of the brightness of each candidate window segment to obtain a sequence of variance values ​​corresponding to each candidate window segment; The candidate window segment with the smallest variance is selected from each variance value sequence, and the intermediate frame identifier of the candidate window segment is determined as the key frame identifier of the current segment. The key frames are then integrated to obtain the key frame set.

[0024] In this embodiment, the difference feature sequence is statistically analyzed frame by frame. The average difference of all pixels in the luminance component between the current frame and the previous frame is calculated to obtain the average luminance value mean_v (equivalent to the average difference value mentioned above) and the current frame identifier id for the current frame. The average luminance value mean_v is compared with a preset threshold, preferably set to 27. A scene switch is determined to have occurred if and only if the joint condition of mean_v > 27 and current frame identifier id > 29 is met. At the same time, the record item (mean_v, id) and the current frame identifier id are output for subsequent determination as the basis for determining the segment boundary. For the first segment, the segment range is defined as from the first frame to the current frame identifier id that first triggers the switching condition. For subsequent segments, the segment range is defined as the frame after the boundary of the previous segment to the current frame identifier id that triggers the switching condition this time, thus obtaining the frame identifier sequence of the current segment.

[0025] After segment determination, to obtain keyframes for the segment, all differential feature pairs (id, mean_v) accumulated in step S101 within the segment are paired in chronological order to obtain a discrete feature sequence for keyframe selection. To statistically select frames that represent the stable intervals of the segment content, the total number of points in the discrete feature sequence, sum, is first counted, and half of sum is set as the window size, window_size. Starting from the first point of the sequence, continuous point sets are extracted sequentially according to window_size, resulting in multiple candidate window segments of number (sum − window_size). For each candidate window segment, the variance of its internal mean brightness is calculated, obtaining a variance value sequence corresponding one-to-one with the candidate window.

[0026] Furthermore, the candidate window segment with the smallest variance is selected from the variance value sequence and considered as the interval with the most stable change within the segment. The frame identifier corresponding to the middle position of this candidate window segment is determined as the keyframe identifier of the current segment. The above method of determining the minimum variance-middle frame is consistent with the criterion of solving the center point of the smoothest segment on the (id, mean_v) discrete curve, which can reflect the stable representation of brightness changes within the segment at the statistical level. For possible occasional parallel cases, the rule of prioritizing the earlier time or smaller sequence number can be used for disambiguation to ensure the uniqueness of the keyframe identifier. Through the above joint judgment and minimum variance window selection process, the keyframe identifiers of each segment are determined in turn. Finally, the determined keyframe identifiers are integrated to obtain a keyframe set that corresponds one-to-one with the segment, providing a reliable representative frame basis for subsequent keyframe-based cue word construction, condition generation, and color propagation mapping modeling.

[0027] In step S103, prompt words are constructed for the keyframe set to obtain a prompt word set used to drive conditional coloring. First, scene semantic parsing and element extraction are performed on the keyframes to generate initial prompt words that match the screen content. Then, formatting and necessary vocabulary expansion are performed to ensure that the prompt words and keyframes establish a correspondence at the frame identifier level, so that they can be called at the fragment and frame levels subsequently.

[0028] In one embodiment, step S103 includes: The keyframe set is input frame by frame into a preset prompt word generation model to perform scene semantic parsing and element extraction, and to obtain initial prompt words that match the scene of each keyframe. The initial prompt words are appended with fixed phrases one by one and then concatenated in a uniform format to obtain the extended prompt words; Establish a correspondence between the extended prompt words and the keyframe set according to frame identifiers, and output a set of prompt words used to drive conditional coloring.

[0029] In this embodiment, for a given set of keyframes, each frame is input sequentially into the image-to-text module of a preset prompt word generation model (e.g., an image-to-text module based on promptgen_base_v2.0) according to its frame identifier. This module performs scene semantic parsing and element extraction on each keyframe, outputting initial prompt words that match the semantics of that frame. The semantic parsing preferably includes the identification of elements such as the main object, scene environment, posture and actions, structural relationships, and material details, and obtains initial prompt words in a phrase format, which are then cached one-to-one with the corresponding keyframe identifiers. The initial prompt words are then standardized and expanded, specifically by appending fixed phrases such as "color," "photographic level," "high definition," and "realistic" to the end of each initial prompt word to enhance the imaging style and detail fidelity. After appending, the words are uniformly formatted and concatenated, including merging synonyms, removing redundant words, unifying separators and punctuation, and rearranging the word order according to the subject-environment-detail-style sequence, to obtain expanded prompt words corresponding to each keyframe. To avoid excessive differences in cue word length between different keyframes, a maximum length threshold can be set, and the excess portion can be truncated and retained based on semantic importance. A correspondence between cue words and keyframe identifiers is established, pairing the frame identifier of each keyframe with its extended cue word to obtain a mapping entry from frame identifier to extended cue word. All entries are then summarized in chronological order to generate a cue word set used to drive conditional coloring.

[0030] In step S104, conditional image generation is performed using a set of prompt words to obtain a set of colorized keyframes that correspond one-to-one with the keyframe set. An image generation configuration object with conditional control capabilities can be loaded, and the structural information of the keyframes and the corresponding prompt words are used as combined inputs to perform controlled generation to obtain color candidate images. Colorization of the keyframes is completed while preserving the original structural details, and finally integrated into a set of colorized keyframes for the fragment, which is used for subsequent reference and dissemination.

[0031] In one embodiment, step S104 includes: Based on the set of prompt words, a preset diffusion generation model is loaded as the base model, and a preset control network is loaded as the constraint module to obtain a generation configuration object. By using the frame-by-frame calculation of the edge map and depth map of the keyframe set, the edge map and depth map of each keyframe are bound to the corresponding prompt words to obtain a conditional input group organized by frame identifier; The condition input group is input into the generation configuration object for generation processing to obtain a controlled generation instruction set; According to the controlled generation instruction set, image-to-image diffusion generation is performed with the corresponding key frame as the reference frame, and the color candidate image corresponding to each key frame is output. The color candidate images are processed to preserve detail, and then integrated to obtain a set of colorized keyframes.

[0032] In this embodiment, based on the output cue word set, a preset diffusion generation model is loaded as a base model, and a preset control network is loaded as a constraint module to obtain a generation configuration object. Preferably, the base model adopts an open-source diffusion model with detail generation capabilities (such as RealVisXL_V_0_Lighting), and the control network adopts two types of control branches (such as Canny and Depth) to inject structural and geometric constraint signals into the diffusion process during generation, ensuring that the generated result is consistent with the input keyframes in terms of contour, depth of field, and detail. The control network and the diffusion model work together, and its core is to inject external control signals such as edges, pose, and depth into the diffusion generation process, thereby achieving controlled generation and preserving the creative generation capabilities of the base model. Then, the structural prior is calculated frame by frame for the keyframe set, generating edge maps and depth maps for each keyframe, and binding them with the corresponding cue words to form a conditional input group organized by frame identifier. The prompts are derived from the semantic parsing and element extraction of keyframe images by the graph-to-text module, and fixed phrases such as "color, photographic quality, high definition, and realistic" are appended to the end to unify the imaging style and detail requirements. These prompts, as semantic conditions, together with the edge map and depth map, constitute a multi-condition driven input for subsequent controlled generation.

[0033] Furthermore, the conditional input group organized by frame identifier is input into the generation configuration object to perform controlled generation, resulting in a controlled generation instruction set corresponding to each keyframe. In the specific implementation, color results are output by explicitly including color keywords in the prompts, while the control network constrains the structure and details, ensuring that the generation process uses the keyframe image as a reference for image-to-image diffusion generation, avoiding deviation from the original geometry and texture. According to the controlled generation instruction set, image-to-image diffusion generation is performed on each keyframe, outputting color candidate images. To ensure that the structure of the reference frame is not destroyed, detail preservation processing is performed on the color candidate images. That is, during the generation stage, the control network constrains the consistency of contour and depth, and during the output stage, the consistency of edges and textures is checked, preferably retaining geometric details consistent with the original keyframe, and coloring is only performed in the chroma domain, thereby achieving the processing goal of coloring only the reference frame without changing the details of the original input image. If necessary, threshold and confidence rules can be used to backtrack or slightly correct areas that do not meet the consistency requirements to ensure consistency with the keyframe structure.

[0034] Finally, all detail-preserving color candidate images are aggregated and integrated according to keyframe identifiers to obtain a set of colorized keyframes. This set serves as the reference input for subsequent color propagation and temporal synthesis within segments, providing stable color priors for non-keyframes while preserving the structure and details of keyframes.

[0035] In step S105, the reference relationship within the segment is modeled based on the set of colorization keyframes, and a color propagation mapping is established in conjunction with the frame sequence position within the segment. Based on the multi-source features of keyframes, historical colorized frames, and frames to be processed, representations are established for the luminance and chrominance channels respectively, and feature alignment and mapping generation are performed under the reference relationship. This enables the color propagation mapping to have both reference frame guidance and temporal constraint capabilities, providing a stable mapping relationship for subsequent frame-by-frame applications.

[0036] In step S106, the non-key frames within the segment are applied frame-by-frame based on the color propagation mapping. First, the luminance features of the frame to be colored are calculated and matched with historical reference features. Then, the chromaticity representation of the target frame is obtained in the feature space according to the mapping, and fused with the luminance component of that frame to generate a colored image. Finally, the colored results of the non-key frames are temporally synthesized with the colored key frames according to the frame identifier order to output the colored video corresponding to the segment. This frame-by-frame processing completes the fusion and reconstruction of luminance and chromaticity at the channel level and outputs the video sequence in chronological order.

[0037] In one embodiment, step S106 includes: The set of coloring keyframes is used as a reference sample set for color propagation within the segment to obtain the reference sample set; The reference sample set is subjected to multi-frame fusion processing to obtain a fused historical feature set; The luminance component features of the non-key frames to be colored are extracted frame by frame, and guided feature extraction processing is performed to obtain the luminance feature matrix of the target frame. Establish a feature alignment relationship between the target frame luminance feature matrix and the historical feature set to obtain the historical chrominance feature matrix; Based on the color propagation mapping, the distance metric between the target frame luminance feature matrix and the historical chrominance feature matrix is ​​calculated, and then the distance metric is converted into a similarity matrix using the exponential normalization function. The historical chroma feature matrix is ​​weighted and summed using the similarity matrix to obtain the chroma component features corresponding to the target frame; The chromaticity component features are fused with the target frame luminance feature matrix to generate a colorized image corresponding to a non-key frame. The colorized image and the set of colorized keyframes are sequentially synthesized according to the frame identifier order to output the colorized video.

[0038] In this embodiment, the obtained set of colorization keyframes is used as a reference sample set for color propagation within the segment, serving as the benchmark for subsequent calculations. Specifically, the colorization keyframes are used as reference samples (reference frames) for subsequent colorization of the segment, and are input along with previously colorized historical frames for subsequent feature construction and matching. During the reference sample construction stage, to enhance the usable information of color and structure across frames, the reference sample set is input into the multi-frame fusion module (MFP) to fuse the historically colorized frames with the keyframes, obtaining the luminance channel feature matrix and chrominance channel feature matrix respectively. This fusion result serves as a historical feature set, used to establish a stable feature alignment relationship and color guidance between the target frame and the reference samples. In the target frame feature extraction stage, the non-keyframes to be colorized are processed frame by frame. First, the luminance component of the frame is extracted, and then the luminance feature matrix of the target frame is obtained through the guided feature extraction module (PVGFE) to facilitate matching and propagation with the historical feature set within the same representation space. In the feature alignment and similarity calculation stage, based on the pre-established color propagation mapping, the distance metric is calculated between the target frame luminance feature matrix and the historical chrominance feature matrix. Euclidean distance is preferably used to measure the closeness of the two in terms of luminance representation. Subsequently, the obtained distance metric is transformed into a similarity matrix through an exponential normalization function (softmax) to quantify the matching weight of the target frame at each position of the reference sample.

[0039] Furthermore, in the chroma feature solving stage, the historical chroma feature matrix is ​​weighted and summed using a similarity matrix to obtain the chroma component features corresponding to the target frame, realizing controlled propagation of chroma information from the reference sample to the target frame. This process uses the multi-frame fusion result of the reference sample set as the chroma prior, and ensures that the propagation intensity is consistent with the degree of structural matching through similarity weight allocation. In the channel fusion and frame output stage, the obtained chroma component features are fused with the luminance component of the target frame at the channel level to generate the colorized image of the non-key frame; after the current frame is colorized, the result is merged into the historical colorized sequence for subsequent frames to continue to reference and propagate. Finally, the colorized results of the non-key frames are temporally synthesized with the set of colorized key frames according to the time order of the frame identifiers to output the colorized video corresponding to the segment.

[0040] In one embodiment, step S106 further includes: Extract the chroma channel features from the set of historical colored frames to obtain a reference chroma feature set; The reference chroma feature set is input into the local attention module, and attention calculation is performed on the chroma channels between historical frames to extract an attention feature map that indicates historical chroma relationships. The attention feature map is summed pixel by pixel with the brightness feature of the current frame to be colored to obtain the synthesized feature; The synthesized features are decoded and mapped to output the chroma channel features of the current frame to be colored, and channel-level fusion is performed with the corresponding luminance channel to generate the corresponding colored image. The colorized image and the set of colorized keyframes are sequentially synthesized according to the frame identifier order to output the colorized video.

[0041] In this embodiment, a reference chroma feature set is constructed. Historical frames that have already been colored within the segment (including key frames and their subsequent colored frames) are selected. Chroma channel features are extracted from each frame, stacked chronologically, and unified to the same representation dimension to obtain the reference chroma feature set, which is used to represent the chroma priors that evolve over time within the segment. Local attention calculation is performed on the historical chroma channels. The reference chroma feature set is input into the local attention module, and a correlation metric is established for corresponding regions between historical frames in the temporal neighborhood. An attention feature map is calculated to indicate the historical chroma relationships. This attention feature map depicts the chroma correspondence between the same or similar image regions within the segment at different time points and can serve as constraint information for solving the chroma of the current frame. Then, the construction of synthetic features guided by the current frame's luminance is completed. For the current frame to be colored, its luminance guidance features (e.g., obtained from the aforementioned luminance channel processing and guidance feature extraction module) are extracted and summed pixel-by-pixel with the attention feature map according to pixel positions to obtain the synthetic features. The current chroma solution can be expressed by the following formula: [ Y = D, (V + L)] Where D is the similarity matrix calculated based on the luminance features of the current frame and the prior luminance structure of the historical frame, V is the historical chrominance prior term, L is the attention feature map based on the chrominance relationship of the historical frame, and Y is the chrominance channel feature to be obtained in the current frame.

[0042] Furthermore, the synthesized features are decoded and mapped to output the chroma channel features of the current frame. Specifically, a decoder compatible with color propagation mapping can be used to perform nonlinear mapping and reconstruction of the synthesized features, outputting a chroma channel result consistent with the resolution of the current frame. This chroma channel is then fused with the luminance channel of the current frame at the channel level to generate a colorized image corresponding to the current non-key frame. After completion, this colorized image is incorporated into the historical colorized sequence, continuously providing updated chroma and attention priors for subsequent frames.

[0043] Finally, the non-keyframe colorized images and the set of colorized keyframes are sequentially synthesized according to frame identifier order to output the colorized video corresponding to the segment. By introducing historical chroma attention feature maps and solving them under the constraints of the similarity matrix, the above process utilizes the chroma priors of keyframes and historical frames, while combining the global context and temporal relationship of the video to achieve stable propagation of the chroma results of the current frame, reducing color flicker caused by inconsistencies across frames.

[0044] Combination Figure 2 As shown, Figure 2 This is a schematic block diagram of a black and white video colorization device provided in an embodiment of the present invention. The black and white video colorization device 200 includes: Image extraction unit 201 is used to extract the luminance component of the color space and calculate the difference between adjacent frames for the input black and white video frames respectively, so as to obtain the difference feature sequence and the corresponding frame identifier sequence used to describe the inter-frame changes. The threshold determination unit 202 is used to perform threshold detection and determination on the differential feature sequence, and divide the frame identifier sequence into multiple segments based on the determination result to determine the key frame set of each segment; The prompt construction unit 203 is used to construct prompt words using the keyframe set to obtain a prompt word set for driving conditional coloring; Image generation unit 204 is used to generate a conditional image using the set of prompt words to obtain a set of colorized keyframes corresponding to the set of keyframes; Color mapping unit 205 is used to model intra-segment reference relationships based on the set of coloring keyframes and establish color propagation mapping by combining the frame sequence positions of each segment. The video output unit 206 is used to perform frame-by-frame application based on the color propagation mapping, complete the coloring of non-key frames within the segment, and synthesize them in chronological order to obtain a colorized video.

[0045] In this embodiment, the image extraction unit 201 extracts the luminance component of the color space and calculates the difference between adjacent frames for the input black and white video frames to obtain a difference feature sequence describing the inter-frame changes and a corresponding frame identifier sequence; the threshold determination unit 202 performs threshold detection and determination on the difference feature sequence, and divides the frame identifier sequence into multiple segments based on the determination result to determine the key frame set of each segment; the prompt construction unit 203 uses the key frame set to construct prompt words to obtain a prompt word set for driving conditional colorization; the image generation unit 204 uses the prompt word set to generate a conditional image to obtain a colorization key frame set corresponding to the key frame set; the color mapping unit 205 models the intra-segment reference relationship based on the colorization key frame set, and establishes a color propagation mapping by combining the frame sequence position of each segment; the video output unit 206 applies the color propagation mapping frame by frame to complete the colorization of non-key frames within the segment and synthesizes them in chronological order to obtain a colorized video.

[0046] In one embodiment, the image extraction unit 201 is specifically used for: The input black and white video is processed in chronological order to establish a frame identifier that corresponds one-to-one with time. The frame identifier is converted frame by frame into a representation of hue-saturation-luminance, and luminance components are extracted from each frame to obtain a luminance component sequence. By performing pixel-by-pixel difference calculation between the current frame and the previous frame in the brightness component sequence, the difference image corresponding to the current frame is obtained; Perform full-pixel statistics on the difference image to obtain a difference metric value used to measure the intensity of change between the current frame and the previous frame; The difference metric value is paired and stored with the corresponding current frame identifier to obtain a difference feature pair; The difference features are accumulated in frame identifier order to generate a difference feature sequence and a corresponding frame identifier sequence.

[0047] In one embodiment, the threshold determination unit 202 is specifically used for: Based on the difference feature sequence, the difference in luminance components between the current frame and the previous frame is statistically analyzed for all pixels to obtain the average luminance value and the current frame identifier corresponding to the current frame. The average brightness value is compared with a preset threshold, and combined with the current frame identifier for joint determination to obtain the frame identifier sequence of the current segment; The frame identifier sequences are paired in chronological order to obtain discrete feature sequences for keyframe filtering; The total number of points in the discrete feature sequence is counted, and half of the total number of points is set as the window size. Starting from the first point of the sequence, continuous point sets are extracted sequentially according to the window size to obtain multiple candidate window segments. Calculate the mean variance of the brightness of each candidate window segment to obtain a sequence of variance values ​​corresponding to each candidate window segment; The candidate window segment with the smallest variance is selected from each variance value sequence, and the intermediate frame identifier of the candidate window segment is determined as the key frame identifier of the current segment. The key frames are then integrated to obtain the key frame set.

[0048] In one embodiment, the prompt construction unit 203 is specifically used for: The keyframe set is input frame by frame into a preset prompt word generation model to perform scene semantic parsing and element extraction, and to obtain initial prompt words that match the scene of each keyframe. The initial prompt words are appended with fixed phrases one by one and then concatenated in a uniform format to obtain the extended prompt words; Establish a correspondence between the extended prompt words and the keyframe set according to frame identifiers, and output a set of prompt words used to drive conditional coloring.

[0049] In one embodiment, the image generation unit 204 is specifically used for: Based on the set of prompt words, a preset diffusion generation model is loaded as the base model, and a preset control network is loaded as the constraint module to obtain a generation configuration object. By using the frame-by-frame calculation of the edge map and depth map of the keyframe set, the edge map and depth map of each keyframe are bound to the corresponding prompt words to obtain a conditional input group organized by frame identifier; The condition input group is input into the generation configuration object for generation processing to obtain a controlled generation instruction set; According to the controlled generation instruction set, image-to-image diffusion generation is performed with the corresponding key frame as the reference frame, and the color candidate image corresponding to each key frame is output. The color candidate images are processed to preserve detail, and then integrated to obtain a set of colorized keyframes.

[0050] In one embodiment, the video output unit 206 is specifically used for: The set of coloring keyframes is used as a reference sample set for color propagation within the segment to obtain the reference sample set; The reference sample set is subjected to multi-frame fusion processing to obtain a fused historical feature set; The luminance component features of the non-key frames to be colored are extracted frame by frame, and guided feature extraction processing is performed to obtain the luminance feature matrix of the target frame. Establish a feature alignment relationship between the target frame luminance feature matrix and the historical feature set to obtain the historical chrominance feature matrix; Based on the color propagation mapping, the distance metric between the target frame luminance feature matrix and the historical chrominance feature matrix is ​​calculated, and then the distance metric is converted into a similarity matrix using the exponential normalization function. The historical chroma feature matrix is ​​weighted and summed using the similarity matrix to obtain the chroma component features corresponding to the target frame; The chromaticity component features are fused with the target frame luminance feature matrix to generate a colorized image corresponding to a non-key frame. The colorized image and the set of colorized keyframes are sequentially synthesized according to the frame identifier order to output the colorized video.

[0051] In one embodiment, the video output unit 206 is further specifically used for: Extract the chroma channel features from the set of historical colored frames to obtain a reference chroma feature set; The reference chroma feature set is input into the local attention module, and attention calculation is performed on the chroma channels between historical frames to extract an attention feature map that indicates historical chroma relationships. The attention feature map is summed pixel by pixel with the brightness feature of the current frame to be colored to obtain the synthesized feature; The synthesized features are decoded and mapped to output the chroma channel features of the current frame to be colored, and channel-level fusion is performed with the corresponding luminance channel to generate the corresponding colored image. The colorized image and the set of colorized keyframes are sequentially synthesized according to the frame identifier order to output the colorized video.

[0052] Since the embodiments of the apparatus and the embodiments of the method correspond to each other, please refer to the description of the embodiments of the method for the embodiments of the apparatus, which will not be repeated here.

[0053] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed, can perform the steps provided in the above embodiments. The storage medium may include various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0054] This invention also provides a computer device, which may include a memory and a processor. The memory stores a computer program, and when the processor calls the computer program in the memory, it can implement the steps provided in the above embodiments. Of course, the computer device may also include various network interfaces, a power supply, a graphics card, etc., to utilize the graphics card's performance to operate the model, such as for inference and training.

[0055] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to in the method section. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of the claims of this application.

[0056] It should also be noted that, in this specification, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A method of coloring a black and white video, characterized by, The method comprises the following steps: Respectively extracting the luminance component of the color space of the input black and white video frames and calculating the difference between adjacent frames to obtain a difference feature sequence for describing the inter-frame changes and a corresponding frame identification sequence; Threshold detection and judgment are performed on the difference feature sequence, and based on the judgment result, the frame identification sequence is divided into multiple segments to determine the key frame set of each segment; Prompt word construction is performed using the key frame set to obtain a prompt word set for driving conditional coloring; Conditional image generation is performed using the prompt word set to obtain a coloring key frame set corresponding to the key frame set; Intra-segment reference relationship modeling is performed according to the coloring key frame set, and combined with the frame sequence position of each segment, a color propagation mapping is established; Based on the color propagation mapping, frame-by-frame application is performed to complete the coloring of non-key frames within the segment and to synthesize in chronological order to obtain a colored video.

2. The black and white video coloring method according to claim 1, characterized by, The method comprises the following steps: The input black and white video is obtained in chronological order to establish a one-to-one correspondence between the frame identification and the time; The frame identification is converted into a hue-saturation-luminance three-component representation form frame by frame, and the luminance component image is extracted from each frame to obtain a luminance component sequence; Pixel-by-pixel difference calculation is performed on the current frame and the previous frame in the luminance component sequence to obtain a difference image corresponding to the current frame; Full-pixel statistics are performed on the difference image to obtain a difference metric value for measuring the change intensity of the current frame and the previous frame; The difference metric value and the corresponding current frame identification are paired and stored to obtain a difference feature pair; The difference feature pairs are accumulated in the order of frame identification to generate a difference feature sequence and a corresponding frame identification sequence.

3. The black and white video coloring method of claim 1, wherein, The method comprises the following steps: Full-pixel statistics are performed on the luminance component difference between the current frame and the previous frame based on the difference feature sequence to obtain the luminance mean value of the current frame and the current frame identification; The luminance mean value is compared with the preset threshold value, and a joint determination is made in combination with the current frame identification to obtain the frame identification sequence of the current segment; The frame identification sequence is paired in chronological order to obtain a discrete feature sequence for key frame screening; The total number of points in the discrete feature sequence is counted, and half of the total number of points is set as the window size. Continuous point sets are extracted in sequence from the first point of the sequence with the window size to obtain multiple candidate window segments; The luminance mean value variance of each candidate window segment is calculated to obtain a variance value sequence corresponding to each candidate window segment; The candidate window segment with the smallest variance is selected from each variance value sequence, and the middle frame identification of the candidate window segment is determined as the key frame identification of the current segment to obtain a key frame set.

4. The black and white video coloring method of claim 1, wherein, The method comprises the following steps: The key frame set is input frame by frame into a preset prompt word generation model to perform scene semantic analysis and element extraction, and initial prompt words matched with each key frame scene are obtained; The initial prompt words are appended with fixed phrases and uniformly formatted spliced to obtain extended prompt words; The extended prompt words are matched with the key frame set according to frame identifiers, and a prompt word set for driving conditional coloring is output.

5. The black and white video coloring method of claim 1, wherein, The conditional image generation is performed by using the prompt word set to obtain a set of colored key frames corresponding to the key frame set, which comprises: A preset diffusion type generation model is loaded as a base model based on the prompt word set, and a preset control network is loaded as a constraint module to obtain a generation configuration object; The edge map and the depth map of the key frame set are calculated frame by frame, and the edge map and the depth map of each key frame are bound with the corresponding prompt word to obtain a conditional input group organized according to frame identifiers; The conditional input group is input into the generation configuration object for generation processing to obtain a controlled generation instruction set; According to the controlled generation instruction set, image-to-image diffusion type generation is performed with the corresponding key frame as a reference frame to output a color candidate image corresponding to each key frame; Detail preservation processing is performed on the color candidate image to obtain a set of colored key frames.

6. The black and white video coloring method of claim 1, wherein, The color propagation mapping is applied frame by frame to complete the coloring of non-key frames within a segment and to synthesize in time sequence to obtain a colored video, which comprises: The set of colored key frames is taken as a reference sample set for color propagation within a segment to obtain a reference sample set; Multi-frame fusion processing is performed on the reference sample set to obtain a fused historical feature set; A non-key frame to be colored is extracted frame by frame to obtain a target frame brightness feature matrix; The target frame brightness feature matrix is matched with the historical feature set to obtain a historical chroma feature matrix; Distance measurement between the target frame brightness feature matrix and the historical chroma feature matrix is calculated based on the color propagation mapping, and the distance measurement is converted into a similarity matrix by using an exponential normalization function; The historical chroma feature matrix is weighted and summed by using the similarity matrix to obtain a chroma component feature corresponding to the target frame; The chroma component feature is fused with the target frame brightness feature matrix to generate a colored image corresponding to the non-key frame; The colored image and the set of colored key frames are synthesized in time sequence according to the frame identifier sequence to output a colored video.

7. The black and white video coloring method of claim 1, wherein, The color propagation mapping is applied frame by frame to complete the coloring of non-key frames within a segment and to synthesize in time sequence to obtain a colored video, which further comprises: Chroma channel features of a set of historical colored frames are extracted to obtain a reference chroma feature set; The reference chroma feature set is input into a local attention module to perform attention calculation on the chroma channel between historical frames to extract an attention feature map indicating historical chroma relationships; The attention feature map is pixel by pixel summed with the brightness feature of the current frame to be colored to obtain a synthesized feature; The synthesis feature is subjected to a decoding mapping process, and a chroma channel feature of the current frame to be colored is outputted and subjected to channel-level fusion with a corresponding luminance channel to generate a corresponding colored image; The colored image is time-synthesized with the set of colored key frames in the order of the frame identifiers, and a colored video is outputted.

8. A black and white video colorizing apparatus characterized by comprising: The method comprises the following steps: A picture extraction unit is configured to extract luminance components of color spaces of inputted black-and-white video frames and calculate adjacent frame differences respectively to obtain a difference feature sequence for describing inter-frame changes and a corresponding frame identifier sequence; A threshold determination unit is configured to perform threshold detection determination on the difference feature sequence and divide the frame identifier sequence into multiple segments based on the determination result to determine a key frame set of each segment; A prompt construction unit is configured to perform prompt word construction using the key frame set to obtain a prompt word set for driving conditional coloring; An image generation unit is configured to perform conditional image generation using the prompt word set to obtain a set of coloring key frames corresponding to the key frame set; A color mapping unit is configured to model reference relationships within segments according to the set of coloring key frames, combine frame sequence positions of each segment, and establish color propagation mapping; A video output unit is configured to perform frame-by-frame application based on the color propagation mapping, complete coloring of non-key frames within segments, and time-synthesize to obtain a colored video.

9. A computer device, comprising: The computer program is stored on the computer readable storage medium and is executable on the processor, and the processor executes the computer program to implement the black-and-white video coloring method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer program is stored on the computer readable storage medium and is executable on the processor, and the processor executes the computer program to implement the black-and-white video coloring method according to any one of claims 1 to 7.

Images