Multimedia data processing method and device, computer device and storage medium

Abstract

The application relates to a multimedia data processing method and device, computer equipment and a storage medium, and relates to the field of artificial intelligence. The method comprises the following steps: determining a plurality of image sequences according to to-be-processed multimedia data, the image sequences comprising N target images arranged in time sequence, wherein N is a positive integer; for any image sequence, determining a pixel classification graph corresponding to each target image in the image sequence, the pixel classification graph comprising classification features of each pixel in the target image, the classification features being used for characterizing an object to which the pixel belongs; and performing coloring processing on each target image in the image sequence and the pixel classification graph corresponding to each target image according to a coloring model constructed based on a transformer, to obtain a color image sequence corresponding to the image sequence; and obtaining target multimedia data corresponding to the to-be-processed multimedia data according to the color image sequence corresponding to each image sequence. The method can improve the definition of the colored video.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a multimedia data processing method, apparatus, computer equipment, and storage medium. Background Technology

[0002] With the rapid development of technology, numerous cameras are deployed near bank branches and ATMs (Automatic Teller Machines) in people's daily lives, especially in the financial sector, to ensure the safety of people's assets. Currently, most banks use color cameras, which can capture more image information, such as clothing colors and object colors, resulting in clearer images. However, in low-light conditions at night, the night vision mode of these cameras captures black and white videos. Due to the lack of color, the video clarity decreases, posing a certain security risk.

[0003] To address these issues, video colorization technology has emerged. Traditional video colorization methods extract a specific target and perform a series of processing steps on the target to achieve better colorization. However, processing only the target without addressing the background color can lead to uneven edges between the colored target and the background, target distortion, and consequently, a decrease in the clarity of the video image. Summary of the Invention

[0004] Therefore, it is necessary to provide a multimedia data processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product capable of accurately colorizing, in response to the aforementioned technical problems.

[0005] Firstly, this application provides a multimedia data processing method. The method includes:

[0006] Based on the multimedia data to be processed, multiple image sequences are determined, wherein the image sequences include N target images arranged in chronological order, where N is a positive integer;

[0007] For any of the image sequences, a pixel classification map corresponding to each target image in the image sequence is determined. The pixel classification map includes the classification features of each pixel in the target image, and the classification features are used to characterize the object to which the pixel belongs.

[0008] Based on the coloring model built on transformer, coloring processing is performed on each target image and the pixel classification map corresponding to each target image in the image sequence to obtain the color image sequence corresponding to the image sequence;

[0009] The target multimedia data corresponding to the multimedia data to be processed is obtained based on the color image sequence corresponding to each of the image sequences.

[0010] In one embodiment, the colorization model includes a temporal branch and N spatial branches. The step of colorizing each target image and its corresponding pixel classification map in the image sequence according to the colorization model built based on a transformer, to obtain a color image sequence corresponding to the image sequence, includes:

[0011] For any of the target images, the target image and the pixel classification map corresponding to the target image are input into any of the spatial domain branches, wherein different target images correspond to different spatial domain branches;

[0012] Feature extraction is performed on the target image through the spatial domain branch to obtain the feature image corresponding to the target image, and the target spatial domain features corresponding to the target image are extracted based on the feature image corresponding to the target image and the pixel classification map.

[0013] By performing temporal feature extraction on the feature image corresponding to each target image through the temporal branch, the target temporal features between each target image are obtained;

[0014] Based on the target spatial domain features and the target temporal domain features, a color image corresponding to the target image is reconstructed;

[0015] Based on the color images corresponding to each target image, a color image sequence corresponding to the image sequence is obtained.

[0016] In one embodiment, the spatial domain branch includes a convolution module, a feature fusion module, and an attention unit. The step of extracting features from the target image through the spatial domain branch to obtain a feature image corresponding to the target image, and extracting target spatial domain features corresponding to the target image based on the feature image and the pixel classification map, includes:

[0017] The convolution module is used to extract features from the target image to obtain the feature image corresponding to the target image;

[0018] The feature fusion module performs feature fusion processing on the feature image and the pixel classification map corresponding to the target image to obtain spatial domain features.

[0019] The spatial domain features of the target image are obtained by performing feature optimization on the spatial domain features through the attention unit.

[0020] In one embodiment, the temporal branch includes a feature fusion module and an attention unit. The step of extracting temporal features from the feature images corresponding to each target image through the temporal branch to obtain target temporal features among the target images includes:

[0021] The feature fusion module performs feature fusion processing on the feature images corresponding to each target image to obtain the temporal features between the target images.

[0022] The attention unit is used to optimize the temporal features to obtain the target temporal features between the target images.

[0023] In one embodiment, the spatial domain branch includes a reconstruction module, wherein reconstructing a color image corresponding to the target image based on the target spatial domain features and the target temporal domain features includes:

[0024] The target spatial domain features and the target temporal domain features are superimposed to obtain the target features corresponding to the target image;

[0025] The target features are reconstructed by the reconstruction module to obtain a color image corresponding to the target image.

[0026] In one embodiment, obtaining the target multimedia data corresponding to the multimedia data to be processed based on the color image sequence corresponding to each of the image sequences includes:

[0027] Get preset multimedia formats;

[0028] By stitching together the color images in the color image sequence in chronological order, a video frame sequence corresponding to the multimedia data to be processed is obtained.

[0029] According to the preset multimedia format, the video frame sequence is converted into target multimedia data corresponding to the multimedia data to be processed.

[0030] In one embodiment, before performing colorization processing on each target image and the corresponding pixel classification map in the image sequence according to the colorization model built based on transformer, to obtain the color image sequence corresponding to the image sequence, the method further includes:

[0031] Retrieve multiple historical color videos;

[0032] Based on the aforementioned historical color videos, determine the color image sequence of each sample, and perform image whitening processing on the color image sequence of each sample to obtain the black and white image sequence of each sample.

[0033] A training set is constructed based on the black and white image sequences of each sample.

[0034] The initial coloring model based on transformer is trained using the training set to obtain a coloring model based on transformer.

[0035] Secondly, this application also provides a multimedia data processing apparatus. The apparatus includes:

[0036] The first determining module is used to determine multiple image sequences based on the multimedia data to be processed, wherein the image sequences include N target images arranged in chronological order, where N is a positive integer;

[0037] The second determining module is used to determine, for any image sequence, a pixel classification map corresponding to each target image in the image sequence, wherein the pixel classification map includes classification features of each pixel in the target image, and the classification features are used to characterize the object to which the pixel belongs;

[0038] The coloring module is used to perform coloring processing on each target image and the pixel classification map corresponding to each target image in the image sequence according to the coloring model built based on transformer, so as to obtain a color image sequence corresponding to the image sequence;

[0039] The conversion module is used to obtain the target multimedia data corresponding to the multimedia data to be processed based on the color image sequence corresponding to each of the image sequences.

[0040] In one embodiment, the coloring model includes a temporal branch and N spatial branches, and the coloring module is further configured to:

[0041] For any of the target images, the target image and the pixel classification map corresponding to the target image are input into any of the spatial domain branches, wherein different target images correspond to different spatial domain branches;

[0042] Feature extraction is performed on the target image through the spatial domain branch to obtain the feature image corresponding to the target image, and the target spatial domain features corresponding to the target image are extracted based on the feature image corresponding to the target image and the pixel classification map.

[0043] By performing temporal feature extraction on the feature image corresponding to each target image through the temporal branch, the target temporal features between each target image are obtained;

[0044] Based on the target spatial domain features and the target temporal domain features, a color image corresponding to the target image is reconstructed;

[0045] Based on the color images corresponding to each target image, a color image sequence corresponding to the image sequence is obtained.

[0046] In one embodiment, the spatial branch includes a convolution module, a feature fusion module, and an attention unit, and the coloring module is further used for:

[0047] The convolution module is used to extract features from the target image to obtain the feature image corresponding to the target image;

[0048] The feature fusion module performs feature fusion processing on the feature image and the pixel classification map corresponding to the target image to obtain spatial domain features.

[0049] The spatial domain features of the target image are obtained by performing feature optimization on the spatial domain features through the attention unit.

[0050] In one embodiment, the temporal branch includes a feature fusion module and an attention unit, and the coloring module is further configured to:

[0051] The feature fusion module performs feature fusion processing on the feature images corresponding to each target image to obtain the temporal features between the target images.

[0052] The attention unit is used to optimize the temporal features to obtain the target temporal features between the target images.

[0053] In one embodiment, the spatial branch includes a reconstruction module, and the coloring module is further configured to:

[0054] The target spatial domain features and the target temporal domain features are superimposed to obtain the target features corresponding to the target image;

[0055] The target features are reconstructed by the reconstruction module to obtain a color image corresponding to the target image.

[0056] In one embodiment, the conversion module is further configured to:

[0057] Get preset multimedia formats;

[0058] By stitching together the color images in the color image sequence in chronological order, a video frame sequence corresponding to the multimedia data to be processed is obtained.

[0059] According to the preset multimedia format, the video frame sequence is converted into target multimedia data corresponding to the multimedia data to be processed.

[0060] In one embodiment, before performing color processing on each target image and the corresponding pixel classification map in the image sequence according to the colorization model built based on transformer, to obtain the color image sequence corresponding to the image sequence, the apparatus further includes:

[0061] The acquisition module is used to acquire multiple historical color videos;

[0062] The third determining module is used to determine the color image sequence of each sample based on each of the historical color videos, and to perform image whitening processing on each of the sample color image sequences to obtain a black and white image sequence of each sample.

[0063] The component module is used to construct a training set based on the black and white image sequences of each sample;

[0064] The training module is used to train the initial coloring model based on the transformer according to the training set, so as to obtain the coloring model based on the transformer.

[0065] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0066] Based on the multimedia data to be processed, multiple image sequences are determined, wherein the image sequences include N target images arranged in chronological order, where N is a positive integer;

[0067] For any of the image sequences, a pixel classification map corresponding to each target image in the image sequence is determined. The pixel classification map includes the classification features of each pixel in the target image, and the classification features are used to characterize the object to which the pixel belongs.

[0068] Based on the coloring model built on transformer, coloring processing is performed on each target image and the pixel classification map corresponding to each target image in the image sequence to obtain the color image sequence corresponding to the image sequence;

[0069] The target multimedia data corresponding to the multimedia data to be processed is obtained based on the color image sequence corresponding to each of the image sequences.

[0070] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0071] Based on the multimedia data to be processed, multiple image sequences are determined, wherein the image sequences include N target images arranged in chronological order, where N is a positive integer;

[0072] For any of the image sequences, a pixel classification map corresponding to each target image in the image sequence is determined. The pixel classification map includes the classification features of each pixel in the target image, and the classification features are used to characterize the object to which the pixel belongs.

[0073] Based on the coloring model built on transformer, coloring processing is performed on each target image and the pixel classification map corresponding to each target image in the image sequence to obtain the color image sequence corresponding to the image sequence;

[0074] The target multimedia data corresponding to the multimedia data to be processed is obtained based on the color image sequence corresponding to each of the image sequences.

[0075] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0076] Based on the multimedia data to be processed, multiple image sequences are determined, wherein the image sequences include N target images arranged in chronological order, where N is a positive integer;

[0077] For any of the image sequences, a pixel classification map corresponding to each target image in the image sequence is determined. The pixel classification map includes the classification features of each pixel in the target image, and the classification features are used to characterize the object to which the pixel belongs.

[0078] Based on the coloring model built on transformer, coloring processing is performed on each target image and the pixel classification map corresponding to each target image in the image sequence to obtain the color image sequence corresponding to the image sequence;

[0079] The target multimedia data corresponding to the multimedia data to be processed is obtained based on the color image sequence corresponding to each of the image sequences.

[0080] The aforementioned multimedia data processing method, apparatus, computer equipment, storage medium, and computer program product, based on the multimedia data to be processed, determine multiple image sequences, each image sequence including N target images arranged in chronological order, where N is a positive integer. For any image sequence, determine the pixel classification map corresponding to each target image in the image sequence. The pixel classification map includes the classification features of each pixel in the target image, which are used to characterize the object to which the pixel belongs. Then, based on a colorization model constructed using transformers, colorize each target image and its corresponding pixel classification map in the image sequence to obtain a color image sequence corresponding to the image sequence. Based on the color image sequences corresponding to each image sequence, obtain the target multimedia data corresponding to the multimedia data to be processed. Based on the aforementioned multimedia data processing method, apparatus, computer equipment, storage medium, and computer program product, the pixel classification map corresponding to each target image is determined. The pixel classification map can represent the object to which each pixel in its corresponding target image belongs, thereby segmenting the objects in the target image at the pixel level. Then, the target image and its corresponding pixel classification map are colorized using a colorization model constructed using transformers. In the above processing, the color model processes the pixels of each object contained in the target image, avoiding the deformation of objects caused by processing only the target object. The edges of different objects in the final color image are smoother, thereby improving the clarity of the video image. Attached Figure Description

[0081] Figure 1 This is a flowchart illustrating a multimedia data processing method in one embodiment;

[0082] Figure 2 This is a flowchart illustrating a multimedia data processing method in another embodiment;

[0083] Figure 3 This is a schematic diagram of the structure of a coloring model built based on transformer in one embodiment;

[0084] Figure 4 This is a flowchart illustrating a multimedia data processing method in another embodiment;

[0085] Figure 5 This is a flowchart illustrating a multimedia data processing method in another embodiment;

[0086] Figure 6 This is a flowchart illustrating a multimedia data processing method in another embodiment;

[0087] Figure 7 This is a flowchart illustrating a multimedia data processing method in another embodiment;

[0088] Figure 8 This is a flowchart illustrating a multimedia data processing method in another embodiment;

[0089] Figure 9 This is a structural block diagram of a multimedia data processing device in one embodiment;

[0090] Figure 10 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0091] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0092] In one embodiment, such as Figure 1 As shown, a multimedia data processing method is provided. This embodiment illustrates the method applied to a terminal. It is understood that this method can also be applied to a server, and further to a system including both a terminal and a server, and is implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0093] Step 102: Based on the multimedia data to be processed, determine multiple image sequences. The image sequences include N target images arranged in chronological order, where N is a positive integer.

[0094] The multimedia data to be processed is the multimedia data to be colored. The multimedia data to be processed can be black and white video or black and white image set.

[0095] Specifically, after acquiring the multimedia data to be processed, preprocessing can be performed based on its content. For example, if the multimedia data is a black and white video, it can be converted into multiple target images; or, if the multimedia data is a set of black and white images, the target images can be sorted according to their capture or creation time. Afterward, data cleaning can be performed on each target image in the multimedia data to be processed, such as removing jagged edges, background noise, and other interference.

[0096] Secondly, the cleaned multimedia data to be processed can be grouped to obtain multiple image sequences. The target images corresponding to the multimedia data to be processed are arranged in chronological order. During the grouping process, starting from the first target image, N adjacent target images can be taken as the first image sequence in chronological order. Then, starting from the (N+1)th target image, N adjacent target images can be taken as the second image sequence, and so on, until all target images corresponding to the multimedia data to be processed are grouped.

[0097] In one embodiment, when the number of target images in the last image sequence is less than N, a corresponding number of target images can be taken forward to ensure that the number of target images in the last image sequence is N. For example, taking a black-and-white video as the multimedia data to be processed, which corresponds to 10 target images, and each image sequence includes 3 target images, the 10 target images corresponding to the black-and-white video are grouped. The last image sequence (image sequence 4) includes only one target image (i.e., image 10). At this point, image sequence 4 is still missing 2 target images. Therefore, two target images can be taken forward in chronological order: image 9 and image 8, forming image sequence 4. In summary, through grouping processing, image sequence 1 is obtained as: image 1, image 2, image 3; image sequence 2 as: image 4, image 5, image 6; image sequence 3 as: image 7, image 8, image 9; and image sequence 4 as: image 8, image 9, image 10.

[0098] Step 104: For any image sequence, determine the pixel classification map corresponding to each target image in the image sequence. The pixel classification map includes the classification features of each pixel in the target image. The classification features are used to characterize the object to which the pixel belongs.

[0099] In this embodiment of the application, for any image sequence, each target image in the image sequence can be segmented by an image segmentation method to obtain the object to which each pixel in the target image belongs (i.e., which object the pixel belongs to), and then determine the pixel classification map corresponding to each target image. The pixel classification map includes the classification features of each pixel in the target image, and the classification features are used to characterize the object to which the pixel belongs.

[0100] In one embodiment, a deep learning-based image segmentation method can be used to process the target image. Specifically, a trained deep convolutional neural network can be used to segment the target image. The target image is input into the deep convolutional neural network, which segments the target image and outputs a pixel classification image corresponding to the target image. In the pixel classification image, each pixel corresponds to a number, which indicates which object the pixel belongs to. The numbers corresponding to pixels belonging to different objects are also different.

[0101] For example, taking the above example again, with Image 1 as the target image, and the objects in Image 1 including a table, chair, apple, grass, and sky as an example, the target image is segmented by a deep convolutional neural network to obtain a pixel classification map. The table, chair, apple, grass, and sky in the target image can be represented by the numbers 0, 1, 2, 3, and 4 respectively. That is, the pixels that make up the table (i.e., the pixels whose object is the table) are all represented by the number 0, the pixels that make up the chair are represented by the number 1, the pixels that make up the apple are represented by the number 2, the pixels that make up the grass are represented by the number 3, and the pixels that make up the sky are represented by the number 4.

[0102] Step 106: Based on the colorization model built on transformer, colorize each target image and the corresponding pixel classification map in the image sequence to obtain the color image sequence corresponding to the image sequence.

[0103] The Transformer model is a neural network model based on the self-attention mechanism. Using machine translation as an example, we can explain the Transformer model. During translation, self-attention more easily captures long-range interdependent features within a sentence. That is, self-attention directly represents the relationship between any two words in a sentence through a single calculation, greatly reducing the distance between distant dependent features and facilitating their effective utilization. Therefore, the Transformer model has the advantage of effectively extracting global dependency features of the input object. In this embodiment, the coloring model built based on the Transformer can effectively extract the dependency features (also known as spatial domain features) between any two pixels in the target image.

[0104] In this embodiment of the application, each target image in an image sequence and the pixel classification map corresponding to the target image can be input into the coloring model. Then, according to the coloring model built based on transformer, the above input data is colorized to obtain the color image of each target image in the image sequence, that is, the color image sequence corresponding to the image sequence is obtained.

[0105] For example, still using the above example, for image sequence 1: image 1, image 2, image 3, classify the pixels of image 1 with the pixels corresponding to image 1. Figure 1 Image 2 and its corresponding pixel classification Figure 2 Image 3 and its corresponding pixel classification Figure 3The images are input into the coloring model. After coloring by the coloring model, the color image 1 corresponding to image 1, the color image 2 corresponding to image 2, and the color image 3 corresponding to image 3 are obtained. That is, the color image sequence 1 corresponding to image sequence 1 includes color image 1, color image 2, and color image 3.

[0106] Step 108: Obtain the target multimedia data corresponding to the multimedia data to be processed based on the color image sequence corresponding to each image sequence.

[0107] In this embodiment of the application, after colorizing multiple image sequences corresponding to the multimedia data to be processed in sequence, the color images in the color image sequences corresponding to each image sequence can be sorted in chronological order to obtain the target multimedia data corresponding to the multimedia data to be processed. The target multimedia data is the colorized color video or color image set.

[0108] In the aforementioned multimedia data processing method, a pixel classification map corresponding to each target image is determined. This pixel classification map represents the object to which each pixel in the target image belongs, allowing for pixel-level segmentation of objects within the target image. Then, a colorization model based on transformers is used to colorize the target image and its corresponding pixel classification map. During this process, the colorization model processes the pixels of each object contained in the target image, avoiding object distortion caused by processing only the target object. The resulting color image exhibits smoother edges between different objects, thereby improving the clarity of the video image.

[0109] Secondly, the coloring model built on transformer can effectively extract the spatial domain features between any two pixels on the same target image, improving the coloring accuracy of the coloring model and thus improving the clarity of the video image.

[0110] In one embodiment, such as Figure 2 As shown, the colorization model includes one temporal branch and N spatial branches. Step 106 involves colorizing each target image and its corresponding pixel classification map in the image sequence based on the transformer-based colorization model to obtain a color image sequence corresponding to the image sequence, including:

[0111] Step 202: For any target image, input the target image and the corresponding pixel classification map into any spatial domain branch, where different target images correspond to different spatial domain branches.

[0112] Among them, reference Figure 3As shown, the coloring model consists of two parts: a temporal branch and a spatial branch, wherein the number of spatial branches is the same as the number of target images in the image sequence.

[0113] In this embodiment of the application, for any target image in the image sequence, the target image and the corresponding pixel classification map are input into any spatial branch in the coloring model, wherein different target images are input into different spatial branches.

[0114] For example, taking the above example again, image sequence 1 includes 3 target images, so the coloring model includes 3 spatial branches. Image 1 and the pixel classification map corresponding to image 1 can be input into spatial branch 1, image 2 and the pixel classification map corresponding to image 2 can be input into spatial branch 2, and image 3 and the pixel classification map corresponding to image 3 can be input into spatial branch 3.

[0115] Step 204: Extract features from the target image through spatial domain branch to obtain the feature image corresponding to the target image, and extract the target spatial domain features corresponding to the target image based on the feature image and pixel classification map.

[0116] In this embodiment of the application, feature extraction of the target image can be performed through spatial domain branching to obtain the feature image corresponding to the target image. Then, the feature image corresponding to the target image and the pixel classification map corresponding to the target image are combined to perform feature extraction, thereby obtaining the target spatial domain features corresponding to the target image. The target spatial domain features can characterize the spatial relationship between pixels in the same target image.

[0117] Step 206: Extract temporal features from the feature images corresponding to each target image through temporal branching to obtain the target temporal features between each target image.

[0118] In this embodiment, each spatial branch extracts features from the target object to obtain the feature image corresponding to each target object. Then, each spatial branch inputs the feature image corresponding to the target image into the temporal branch. By extracting temporal features from the feature image corresponding to each target image through the temporal branch, the target temporal features between each target image can be obtained. The target temporal features can represent the temporal relationship between different target images and, to some extent, reflect the motion of objects in the target image as time changes.

[0119] Step 208: Based on the spatial domain features and temporal domain features of the target, reconstruct the color image corresponding to the target image.

[0120] In this embodiment, image color can be reconstructed based on the target spatial domain features and the target temporal domain features, thereby obtaining the color image corresponding to the target image.

[0121] Step 210: Obtain the color image sequence corresponding to the image sequence based on the color image corresponding to each target image.

[0122] In this embodiment of the application, according to the aforementioned steps, the N spatial branches in the coloring model will output the color images corresponding to the N target images respectively. Based on the color images corresponding to the N target images, the color image sequence corresponding to the image sequence can be obtained.

[0123] In this embodiment, a colorization model based on transformer is used to colorize each target image in the image sequence. The colorization model based on transformer can effectively extract the spatial domain features between any two pixels in the same target image. The colorization model also includes a temporal branch, which extracts the target temporal features between different target images. The target spatial domain features and target temporal domain features are combined for reconstruction to obtain a color image, which effectively improves the colorization accuracy of the colorization model and thus improves the clarity of the video image.

[0124] In one embodiment, such as Figure 4 As shown, the spatial domain branch includes a convolution module, a feature fusion module, and an attention unit. In step 204, feature extraction is performed on the target image through the spatial domain branch to obtain the feature image corresponding to the target image. Based on the feature image and pixel classification map corresponding to the target image, the target spatial domain features corresponding to the target image are extracted, including:

[0125] Step 402: Extract features from the target image using a convolution module to obtain the feature image corresponding to the target image.

[0126] Each spatial branch includes a convolution module, a feature fusion module, and an attention unit, with the attention unit comprising multiple attention modules. In this embodiment, the target image is input into the coloring model, which then inputs it into the convolution module. The convolution module performs convolution processing on the target image to extract the corresponding feature image.

[0127] Step 404: The feature fusion module performs feature fusion processing on the feature image and pixel classification image corresponding to the target image to obtain spatial domain features.

[0128] In this embodiment, the feature fusion module is connected to the convolution module, and the output of the convolution module is the input of the feature fusion module. That is, the feature image corresponding to the target image obtained by the convolution module is input into the feature fusion module, and the pixel classification map corresponding to the target image input into the coloring model is directly input into the feature fusion module. Then, the feature fusion module performs feature fusion processing on the feature image and the pixel classification map corresponding to the target image to obtain the spatial domain features of the target image.

[0129] In existing convolutional neural network (CNN) techniques, convolutional kernels are used to convolve images to obtain corresponding feature images. Different convolutional kernels extract different features from the image. For example, for a face image, a CNN model can extract the shape features of the facial features based on convolutional kernel 1, the color features based on convolutional kernel 2, and the spatial features between the facial features based on convolutional kernel 3. In this embodiment, the convolutional model convolves the target image, and the resulting feature image can represent the spatial features between objects in the target image.

[0130] The feature fusion module performs feature fusion processing on the feature image and the pixel classification map. Feature fusion can comprehensively utilize the feature image and the pixel classification map to obtain new fused features, achieving complementary advantages of multiple features. Among them, a series of feature fusion methods can be used to directly connect the features represented by the feature image and the pixel features represented by the pixel classification map to obtain the spatial domain features of the target image; or, a parallel strategy can be used to combine the feature vectors corresponding to the feature image and the pixel classification map into a composite vector (which can represent spatial domain features). For example, if X represents the feature image and Y represents the pixel classification map, then the combined composite vector is X+iY.

[0131] It should be noted that the specific method of feature fusion processing described above is only an example of the feature fusion method in the embodiments of this application, and this application does not impose any specific limitations on the specific method of feature fusion processing.

[0132] Step 406: The spatial domain features are optimized by the attention unit to obtain the target spatial domain features of the target image.

[0133] In this embodiment, the attention unit is connected to the feature fusion module, and the output of the feature fusion module is the input of the attention unit. That is, the spatial domain features output by the feature fusion module are input into the attention unit. The attention unit optimizes the spatial domain features to obtain the target spatial domain features of the target image. The target spatial domain features can effectively characterize the spatial relationship between different pixels in the target image.

[0134] In this embodiment, any spatial branch of the coloring model extracts the spatial domain features of the target image by combining the feature image and pixel classification map corresponding to the target image, and uses an attention unit to optimize the spatial domain features to obtain the target spatial domain features. The attention unit can effectively extract the spatial domain features between any two pixels on the same target image. Then, the target spatial domain features and target temporal features are reconstructed to obtain a color image, which can effectively improve the coloring accuracy of the coloring model and thus improve the clarity of the video image.

[0135] In one embodiment, refer to Figure 5 As shown, the temporal branch includes a feature fusion module and an attention unit. In step 206, temporal features are extracted from the feature images corresponding to each target image through the temporal branch to obtain the target temporal features between the target images, including:

[0136] Step 502: Through the feature fusion module, feature fusion processing is performed on the feature images corresponding to each target image to obtain the temporal features between the target images.

[0137] In this embodiment, the temporal branch includes a feature fusion module and an attention unit, with the attention unit comprising multiple attention modules. The feature fusion module in the temporal branch is connected to the convolutional modules of the N spatial features. The feature images corresponding to the N target images output from the N convolutional modules are input into the feature fusion module in the temporal branch. Then, the feature fusion module performs feature fusion processing on the feature images corresponding to the N targets to obtain the temporal features between the target images.

[0138] The specific feature fusion method can refer to the content described in the above embodiments. A series of feature fusion methods can be used to directly connect N feature images, or a parallel strategy can be used to combine the feature vectors corresponding to N feature images into a composite vector.

[0139] Step 504: The temporal features are optimized by the attention unit to obtain the target temporal features between the target images.

[0140] In this embodiment, the feature fusion module is connected to the attention unit, and the output of the feature fusion module is the input of the attention unit. Therefore, the temporal features obtained through the feature fusion module are input into the attention unit. The attention unit can optimize the temporal features to obtain the target temporal features between different target images.

[0141] This application embodiment extracts target temporal features between different target images through temporal domain branching, and reconstructs them by combining target spatial domain features and target temporal domain features to obtain a color image, which effectively improves the coloring accuracy of the coloring model and thus improves the clarity of the video image.

[0142] In one embodiment, such as Figure 6 As shown, the spatial domain branch includes a reconstruction module. Step 208 involves reconstructing the color image corresponding to the target image based on the target's spatial domain features and temporal domain features, including:

[0143] Step 602: Overlay the spatial domain features and temporal domain features of the target to obtain the target features corresponding to the target image.

[0144] Both the spatial and temporal features of the target can be represented by a pixel matrix, where each pixel in the target image corresponds to a numerical value. In the spatial features, the numerical value of any pixel represents the spatial relationship between that pixel and other pixels; in the temporal features, the numerical value of any pixel represents the temporal relationship between that pixel and different target images.

[0145] In this embodiment of the application, the target spatial domain features and the target temporal domain features can be superimposed. That is, for any pixel in the target image, the target spatial domain feature value and the target temporal domain feature value of the pixel are added together to obtain the target features corresponding to the target image.

[0146] Step 604: The target features are reconstructed using the reconstruction module to obtain the color image corresponding to the target image.

[0147] In this embodiment of the application, the target features are input into the corresponding reconstruction module, and then the target features can be reconstructed by the reconstruction module to obtain the color image corresponding to the target image.

[0148] In this embodiment, the coloring model built based on transformer can effectively extract the spatial domain features between any two pixels on the same target image. The coloring model also includes a temporal branch, which extracts the target temporal features between different target images. The target spatial domain features and target temporal domain features are combined for reconstruction to obtain a color image, which effectively improves the coloring accuracy of the coloring model and thus improves the clarity of the video image.

[0149] In one embodiment, such as Figure 7 As shown, in step 108, based on the color image sequences corresponding to each image sequence, the target multimedia data corresponding to the multimedia data to be processed is obtained, including:

[0150] Step 702: Obtain the preset multimedia format.

[0151] In this embodiment of the application, a preset multimedia format is obtained. The preset multimedia format can be set by the staff as needed. If the staff does not modify the preset multimedia format, the preset multimedia format is the same as the storage format of the multimedia data to be processed.

[0152] Step 704: The color images in the color image sequence are stitched together in chronological order to obtain the video frame sequence corresponding to the multimedia data to be processed.

[0153] In this embodiment of the application, when the multimedia data to be processed is a black and white video, it is necessary to stitch together each color image in all color image sequences in chronological order to obtain the images corresponding to each video frame of the multimedia data to be processed, that is, to obtain the video frame sequence corresponding to the multimedia data to be processed. At this time, the video frame sequence is still an image arranged in chronological order.

[0154] When the multimedia data to be processed is a set of black and white images, the color images in all color image sequences are sorted in chronological order, and then the color images are stored according to a preset multimedia format to obtain the color image set corresponding to the black and white image set.

[0155] Step 706: Convert the video frame sequence into target multimedia data corresponding to the multimedia data to be processed, according to the preset video format.

[0156] In this embodiment of the application, a video frame sequence can be converted into a playable color video according to a preset video format, that is, the target multimedia data corresponding to the multimedia data to be processed is obtained.

[0157] In one embodiment, a preset storage path can also be obtained, and the target multimedia data can be stored according to the preset storage path. The preset storage path is the same as the storage path of the multimedia data to be processed, and the staff can also adjust the preset storage path according to actual needs.

[0158] In this embodiment, color images are converted and stored directly according to a preset multimedia format and a preset storage path. The steps of colorization, format conversion, and storage of the multimedia data to be processed are uniformly executed by the server or terminal, which is beneficial for managing multimedia data and also makes it easier for staff to view the target multimedia data.

[0159] In one embodiment, such as Figure 8 As shown, before step 106, which involves colorizing each target image and its corresponding pixel classification image in the image sequence according to the colorization model built based on transformer, to obtain the color image sequence corresponding to the image sequence, the method further includes:

[0160] Step 802: Obtain multiple historical color videos.

[0161] In this embodiment of the application, multiple historical color videos can be acquired. These historical color videos are color images captured by a color camera under sufficient lighting conditions.

[0162] Step 804: Based on each historical color video, determine the color image sequence of each sample, and perform image whitening processing on each sample color image sequence to obtain the black and white image sequence of each sample.

[0163] In this embodiment of the application, for any historical color video, the color images corresponding to each video frame of the historical color video are grouped to obtain multiple sample color image sequences. The sample color image sequence includes N sample color images that are adjacent in time order. The specific grouping steps can be referred to the content described in the foregoing embodiment, and will not be repeated here.

[0164] Secondly, image whitening processing can be performed on each sample color image in each sample color image sequence to convert each sample color image into a sample black and white image, that is, to obtain each sample black and white image sequence.

[0165] Step 806: Construct a training set based on the black and white image sequences of each sample.

[0166] In this embodiment of the application, image segmentation processing is first performed on each sample black and white image in the sample black and white image sequence to obtain the sample pixel classification map corresponding to each sample black and white image. Then, a training set is constructed based on each sample black and white image in the sample black and white image sequence and the sample pixel classification map corresponding to each sample black and white image.

[0167] Step 808: Train the initial coloring model based on transformer according to the training set to obtain the coloring model based on transformer.

[0168] In this embodiment of the application, an initial coloring model based on transformer can be trained according to the training set to obtain a coloring model based on transformer.

[0169] In one embodiment, the initial colorization model can be trained by combining perceptual loss, reconstruction loss, edge loss, and color loss. The specific steps are as follows: For any sample black and white image sequence, the initial colorization model is used to colorize each sample black and white image in the sample black and white image sequence to obtain the target color image sequence corresponding to the sample black and white image sequence.

[0170] Then, the perceptual loss between each target color image and its corresponding sample color image in the target color image sequence can be calculated based on the VGG network; the reconstruction loss (pixel loss) between each target color image and its corresponding sample color image can be calculated; the edge loss between each target color image and its corresponding sample color image can be calculated using the Sobel edge detection operator; the texture information of each target color image is blurred according to the Gaussian blur kernel, and the color loss between each target color image and its corresponding sample color image can be calculated.

[0171] The target loss is obtained by weighted summation of the four types of loss data. When the target loss is less than or equal to the preset loss value, the training of the initial coloring model is stopped, and the coloring model is obtained.

[0172] This embodiment uses a combination of perceptual loss, reconstruction loss, edge loss, and color loss as the target loss, and trains the initial coloring model with this loss. This improves the accuracy of the coloring model's perceptual information and optimizes for color and image edges, avoiding inconsistencies in coloring.

[0173] In a specific embodiment, taking the above example as an example, the multimedia data to be processed is a black and white video with 10 frames in total. The 10 frames are grouped into 3 target images in each image sequence, resulting in 4 image sequences. The 4 image sequences are as follows: Image sequence 1: Image 1, Image 2, Image 3; Image sequence 2: Image 4, Image 5, Image 6; Image sequence 3: Image 7, Image 8, Image 9; Image sequence 4: Image 8, Image 9, Image 10.

[0174] Taking image sequence 1 as an example, the target image can be segmented using a deep convolutional neural network. The target image is input into the deep convolutional neural network, which then segments the target image and outputs a pixel classification map corresponding to the target image.

[0175] If each image sequence contains three target images, then the initial coloring model built based on the transformer should also include one temporal branch and three spatial branches. The initial coloring model is trained using the training set to obtain the coloring model. The specific training process can refer to the steps described in the above embodiments, and will not be repeated here.

[0176] Classify the pixels of image 1 in image sequence 1 and the corresponding pixels of image 1. Figure 1 The input is fed into spatial branch 1 to classify image 2 and its corresponding pixels. Figure 2 The input is fed into spatial branch 2 to classify image 3 and its corresponding pixels. Figure 3The input is fed into the spatial branch 3. The coloring model then performs coloring processing on the above target images and pixel classification images to obtain color image 1 corresponding to image 1, color image 2 corresponding to image 2, and color image 3 corresponding to image 3. That is, color image sequence 1 corresponding to image sequence 1 is obtained. The specific steps of the coloring processing are as described in the above embodiment and will not be repeated here.

[0177] Repeating the above steps yields four color image sequences corresponding to the four image sequences. Color image sequence 3 and color image sequence 4 both include color image 8 and color image 9. For color images 8 and 9, either one of the two color images 8 can be chosen as the final color image 8, and either one of the two color images 9 can be chosen as the final color image 9. Alternatively, the average pixel value of each pixel in the two color images 8 can be calculated, and the final color image 8 can be determined based on the average pixel value of each pixel in the two color images 9.

[0178] Finally, the preset video format is obtained. When the staff does not impose specific restrictions on the preset video format, it is the storage format for black and white video. Taking the preset video format as .mp4 as an example, the color images in the four color image sequences are stitched together in chronological order to obtain the video frame sequence corresponding to the black and white video: color image 1, color image 2... color image 10. Then, the video frame sequence is converted into a color video in .mp4 format.

[0179] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0180] Based on the same inventive concept, this application also provides a multimedia data processing apparatus for implementing the multimedia data processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more multimedia data processing apparatus embodiments provided below can be found in the limitations of the multimedia data processing method described above, and will not be repeated here.

[0181] In one embodiment, such as Figure 9 As shown, a multimedia data processing device is provided, including: a first determining module 902, a second determining module 904, a coloring module 906, and a conversion module 908, wherein:

[0182] The first determining module 902 is used to determine multiple image sequences based on the multimedia data to be processed, wherein the image sequences include N target images arranged in chronological order, where N is a positive integer;

[0183] The second determining module 904 is used to determine the pixel classification map corresponding to each target image in any image sequence. The pixel classification map includes the classification features of each pixel in the target image. The classification features are used to characterize the object to which the pixel belongs.

[0184] The coloring module 906 is used to colorize each target image and the corresponding pixel classification map in the image sequence according to the coloring model built based on transformer, so as to obtain the color image sequence corresponding to the image sequence.

[0185] The conversion module 908 is used to obtain the target multimedia data corresponding to the multimedia data to be processed based on the color image sequence corresponding to each image sequence.

[0186] In this embodiment, a pixel classification map corresponding to each target image is determined. This pixel classification map represents the object to which each pixel in the target image belongs, allowing for pixel-level segmentation of objects in the target image. Then, a colorization model based on a transformer is used to colorize the target image and its corresponding pixel classification map. During this process, the colorization model processes the pixels of each object contained in the target image, avoiding object distortion caused by processing only the target object. The resulting color image exhibits smoother edges between different objects, thereby improving the clarity of the video image.

[0187] In one embodiment, the coloring model includes a temporal branch and N spatial branches, and the coloring module 906 is further configured to:

[0188] For any target image, input the target image and its corresponding pixel classification image into any spatial domain branch, where different target images correspond to different spatial domain branches;

[0189] Feature extraction is performed on the target image through spatial domain branching to obtain the feature image corresponding to the target image. Based on the feature image and pixel classification map, the target spatial domain features corresponding to the target image are extracted.

[0190] Temporal features are extracted from the feature images corresponding to each target image by temporal branching to obtain the target temporal features between each target image.

[0191] Based on the spatial and temporal features of the target, a color image corresponding to the target image is reconstructed;

[0192] Based on the color images corresponding to each target image, a color image sequence corresponding to the image sequence is obtained.

[0193] In one embodiment, the spatial branch includes a convolution module, a feature fusion module, and an attention unit, and the coloring module 906 is further configured to:

[0194] The convolution module extracts features from the target image to obtain the feature image corresponding to the target image;

[0195] The feature fusion module performs feature fusion processing on the feature image and pixel classification map corresponding to the target image to obtain spatial domain features;

[0196] The spatial domain features of the target image are obtained by optimizing the spatial domain features through attention units.

[0197] In one embodiment, the temporal branch includes a feature fusion module and an attention unit, and the coloring module 906 is further configured to:

[0198] The feature fusion module performs feature fusion processing on the feature images corresponding to each target image to obtain the temporal features between the target images.

[0199] By optimizing the temporal features through attention units, the target temporal features between each target image are obtained.

[0200] In one embodiment, the spatial branch includes a reconstruction module, and the coloring module 906 is further configured to:

[0201] The spatial domain features and temporal domain features of the target are superimposed to obtain the target features corresponding to the target image.

[0202] The target features are reconstructed using the reconstruction module to obtain the color image corresponding to the target image.

[0203] In one embodiment, the conversion module 908 is further configured to:

[0204] Get preset multimedia formats;

[0205] By stitching together the color images in the color image sequence in chronological order, a video frame sequence corresponding to the multimedia data to be processed is obtained.

[0206] Based on the preset multimedia format, the video frame sequence is converted into the target multimedia data corresponding to the multimedia data to be processed.

[0207] In one embodiment, before colorizing each target image and its corresponding pixel classification map in the image sequence according to a colorization model built based on a transformer, to obtain a color image sequence corresponding to the image sequence, the apparatus further includes:

[0208] The acquisition module is used to acquire multiple historical color videos;

[0209] The third determining module is used to determine the color image sequence of each sample based on each historical color video, and to perform image whitening processing on each sample color image sequence to obtain a black and white image sequence of each sample.

[0210] The component module is used to construct a training set based on the sequence of black and white images of each sample;

[0211] The training module is used to train the initial coloring model based on the transformer based on the training set to obtain the coloring model based on the transformer.

[0212] Each module in the aforementioned multimedia data processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0213] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 10As shown, the computer device includes a processor, memory, and a communication interface connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a multimedia data processing method.

[0214] Those skilled in the art will understand that Figure 10 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0215] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0216] Based on the multimedia data to be processed, multiple image sequences are determined. Each image sequence includes N target images arranged in chronological order, where N is a positive integer.

[0217] For any image sequence, determine the pixel classification map corresponding to each target image in the image sequence. The pixel classification map includes the classification features of each pixel in the target image. The classification features are used to characterize the object to which the pixel belongs.

[0218] Based on the coloring model built on transformer, the target images and their corresponding pixel classification maps in the image sequence are colored to obtain the color image sequence corresponding to the image sequence.

[0219] Based on the color image sequence corresponding to each image sequence, the target multimedia data corresponding to the multimedia data to be processed is obtained.

[0220] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0221] Based on the multimedia data to be processed, multiple image sequences are determined. Each image sequence includes N target images arranged in chronological order, where N is a positive integer.

[0222] For any image sequence, determine the pixel classification map corresponding to each target image in the image sequence. The pixel classification map includes the classification features of each pixel in the target image. The classification features are used to characterize the object to which the pixel belongs.

[0223] Based on the coloring model built on transformer, the target images and their corresponding pixel classification maps in the image sequence are colored to obtain the color image sequence corresponding to the image sequence.

[0224] Based on the color image sequence corresponding to each image sequence, the target multimedia data corresponding to the multimedia data to be processed is obtained.

[0225] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:

[0226] Based on the multimedia data to be processed, multiple image sequences are determined. Each image sequence includes N target images arranged in chronological order, where N is a positive integer.

[0227] For any image sequence, determine the pixel classification map corresponding to each target image in the image sequence. The pixel classification map includes the classification features of each pixel in the target image. The classification features are used to characterize the object to which the pixel belongs.

[0228] Based on the coloring model built on transformer, the target images and their corresponding pixel classification maps in the image sequence are colored to obtain the color image sequence corresponding to the image sequence.

[0229] Based on the color image sequence corresponding to each image sequence, the target multimedia data corresponding to the multimedia data to be processed is obtained.

[0230] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0231] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0232] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0233] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A multimedia data processing method, characterized in that, The method includes: Based on the multimedia data to be processed, multiple image sequences are determined, wherein the image sequences include N target images arranged in chronological order, where N is a positive integer; For any of the image sequences, a pixel classification map corresponding to each target image in the image sequence is determined. The pixel classification map includes the classification features of each pixel in the target image, and the classification features are used to characterize the object to which the pixel belongs. Based on the coloring model built on transformer, coloring processing is performed on each target image and the pixel classification map corresponding to each target image in the image sequence to obtain the color image sequence corresponding to the image sequence; Based on the color image sequence corresponding to each of the image sequences, the target multimedia data corresponding to the multimedia data to be processed is obtained; The colorization model includes one temporal branch and N spatial branches. The step of colorizing each target image and its corresponding pixel classification map in the image sequence according to the colorization model built on a transformer basis to obtain a color image sequence corresponding to the image sequence includes: For any of the target images, the target image and the pixel classification map corresponding to the target image are input into any of the spatial domain branches, wherein different target images correspond to different spatial domain branches; Feature extraction is performed on the target image through the spatial domain branch to obtain the feature image corresponding to the target image, and the target spatial domain features corresponding to the target image are extracted based on the feature image corresponding to the target image and the pixel classification map. By performing temporal feature extraction on the feature image corresponding to each target image through the temporal branch, the target temporal features between each target image are obtained; Based on the target spatial domain features and the target temporal domain features, a color image corresponding to the target image is reconstructed; Based on the color images corresponding to each target image, a color image sequence corresponding to the image sequence is obtained.

2. The method according to claim 1, characterized in that, The spatial domain branch includes a convolution module, a feature fusion module, and an attention unit. The step of extracting features from the target image through the spatial domain branch to obtain a feature image corresponding to the target image, and extracting target spatial domain features corresponding to the target image based on the feature image and the pixel classification map, includes: The convolution module is used to extract features from the target image to obtain the feature image corresponding to the target image; The feature fusion module performs feature fusion processing on the feature image and the pixel classification map corresponding to the target image to obtain spatial domain features. The spatial domain features of the target image are obtained by performing feature optimization on the spatial domain features through the attention unit.

3. The method according to claim 1, characterized in that, The temporal branch includes a feature fusion module and an attention unit. The step of extracting temporal features from the feature images corresponding to each target image through the temporal branch to obtain target temporal features among the target images includes: The feature fusion module performs feature fusion processing on the feature images corresponding to each target image to obtain the temporal features between the target images. The attention unit is used to optimize the temporal features to obtain the target temporal features between the target images.

4. The method according to claim 1, characterized in that, The spatial domain branch includes a reconstruction module, wherein the reconstruction of the color image corresponding to the target image based on the target spatial domain features and the target temporal domain features includes: The target spatial domain features and the target temporal domain features are superimposed to obtain the target features corresponding to the target image; The target features are reconstructed by the reconstruction module to obtain a color image corresponding to the target image.

5. The method according to claim 1, characterized in that, The step of obtaining the target multimedia data corresponding to the multimedia data to be processed based on the color image sequence corresponding to each of the image sequences includes: Get preset multimedia formats; By stitching together the color images in the color image sequence in chronological order, a video frame sequence corresponding to the multimedia data to be processed is obtained. According to the preset multimedia format, the video frame sequence is converted into target multimedia data corresponding to the multimedia data to be processed.

6. The method according to any one of claims 1 to 5, characterized in that, Before performing color processing on each target image and the corresponding pixel classification map in the image sequence according to the colorization model built based on transformer, to obtain the color image sequence corresponding to the image sequence, the method further includes: Retrieve multiple historical color videos; Based on the aforementioned historical color videos, determine the color image sequence of each sample, and perform image whitening processing on the color image sequence of each sample to obtain the black and white image sequence of each sample. A training set is constructed based on the black and white image sequences of each sample. The initial coloring model based on transformer is trained using the training set to obtain a coloring model based on transformer.

7. A multimedia data processing device, characterized in that, The device includes: The first determining module is used to determine multiple image sequences based on the multimedia data to be processed, wherein the image sequences include N target images arranged in chronological order, where N is a positive integer; The second determining module is used to determine, for any image sequence, a pixel classification map corresponding to each target image in the image sequence, wherein the pixel classification map includes classification features of each pixel in the target image, and the classification features are used to characterize the object to which the pixel belongs; The coloring module is used to perform coloring processing on each target image and the pixel classification map corresponding to each target image in the image sequence according to the coloring model built based on transformer, so as to obtain a color image sequence corresponding to the image sequence; The conversion module is used to obtain the target multimedia data corresponding to the multimedia data to be processed based on the color image sequence corresponding to each of the image sequences; The coloring model includes a temporal branch and N spatial branches. The coloring module is also used to input the target image and the pixel classification map corresponding to the target image into any of the spatial branches for any target image, wherein different target images correspond to different spatial branches. Feature extraction is performed on the target image through the spatial domain branch to obtain the feature image corresponding to the target image, and the target spatial domain features corresponding to the target image are extracted based on the feature image corresponding to the target image and the pixel classification map. By performing temporal feature extraction on the feature image corresponding to each target image through the temporal branch, the target temporal features between each target image are obtained; Based on the target spatial domain features and the target temporal domain features, a color image corresponding to the target image is reconstructed; Based on the color images corresponding to each target image, a color image sequence corresponding to the image sequence is obtained.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images