Image color restoration method and program based on deep learning and split-screen reference image

Abstract

The embodiment of the present application relates to the technical field of image processing, and discloses an image color restoration method and program based on deep learning and a split-screen reference image, which comprises the following steps: acquiring multiple images in a video, classifying the multiple images based on scene information of the multiple images to obtain multiple sets of image collections; determining split-screen reference images corresponding to the multiple sets of image collections respectively; extracting features of the multiple images to be restored and the split-screen reference images respectively to obtain associated image features and reference image features; performing feature matching on the reference image features and the associated image features, and performing color restoration on the images to be restored based on the matching result. In this way, the images of different scenes are colored respectively, and the accuracy of image coloring and restoration is improved; by setting the split-screen reference image, a time sequence association is established between the image features of the images to be restored, the coloring and restoration of the images can consider the association between the front and rear images, and the authenticity of the coloring and restoration of the images is ensured.

Description

TECHNICAL FIELD

[0001] The embodiment of the present application relates to the technical field of image processing, in particular to an image color restoration method and program based on deep learning and split-screen reference images. BACKGROUND

[0002] At present, the coloring methods for black-and-white videos mainly include manual coloring, user interaction coloring and full-automatic coloring. Among them, the manual coloring method needs to spend a large amount of time, is low in efficiency and high in cost under the premise of ensuring the coloring effect of the video; the user interaction coloring method, although improved in quality and efficiency compared with the manual coloring method, has the problems of single coloring result and inapplicability to differentiated batch coloring; the full-automatic coloring method, although high in efficiency, has the problems of relatively low authenticity and accuracy of coloring effect. SUMMARY

[0003] In view of the above problems, the embodiment of the present application provides an image color restoration method and program based on deep learning and split-screen reference images, which are used to solve the problem that the coloring method for black-and-white videos in the prior art cannot balance high efficiency and authenticity and accuracy.

[0004] According to an aspect of the embodiment of the present application, an image color restoration method based on deep learning and split-screen reference images is provided, which comprises:

[0005] obtaining multiple frames of images in a video, and classifying the multiple frames of images based on scene information of the multiple frames of images to obtain multiple sets of image collections;

[0006] determining split-screen reference images corresponding to the multiple sets of image collections respectively;

[0007] respectively extracting features of the multiple frames of images to be restored in the image collections and the split-screen reference images to obtain associated image features of the multiple frames of images to be restored and reference image features of the split-screen reference images; the associated image features of the multiple frames of images to be restored have time sequence correlation, and the time sequence correlation corresponds to an order of the multiple frames of images to be restored in the image collections;

[0008] performing feature matching on the reference image features of the split-screen reference images and the associated image features of the images to be restored, and performing color restoration on the images to be restored based on a matching result to obtain target images.

[0009] In an optional implementation, the classification of the multiple frames of images based on the scene information of the multiple frames of images to obtain the multiple sets of image collections comprises:

[0010] determining the scene information of the multiple frames of images based on video cut information in the multiple frames of images;

[0011] Classify images with same scene information into a set of image collections, to obtain multiple sets of image collections.

[0012] In an optional implementation, after classifying multiple frames of images based on scene information of the multiple frames of images to obtain multiple sets of image collections, the method further includes:

[0013] Preprocess images in the multiple sets of image collections respectively, to remove non-target pixels of the images in the sets of image collections; the non-target pixels are pixels in the images that do not belong to a base frame of the images;

[0014] Adjust brightness of the images in the multiple sets of image collections respectively based on a brightness histogram;

[0015] Adjust contrast and saturation of the images in the multiple sets of image collections respectively.

[0016] In an optional implementation, the feature extraction is performed on the multiple frames of to-be-restored images and the split-shot reference image in the set of images respectively to obtain associated image features of the multiple frames of to-be-restored images and reference image features of the split-shot reference image, including:

[0017] Input the split-shot reference image and the multiple frames of to-be-restored images into a first neural network to obtain the reference image features and image features of the multiple frames of to-be-restored images;

[0018] Input the multiple frames of to-be-restored images into a second neural network in sequence according to an order of the to-be-restored images in the set of images to obtain time sequence association of the multiple frames of to-be-restored images.

[0019] Based on the time sequence association of the multiple frames of to-be-restored images, establish time sequence association between the image features of the multiple frames of to-be-restored images to obtain the associated image features.

[0020] In an optional implementation, the feature matching is performed on the reference image features of the split-shot reference image and the associated image features of the to-be-restored images, and color restoration is performed on the to-be-restored images based on a matching result to obtain target images, including:

[0021] Match the reference image features of the split-shot reference image with the image features of the to-be-restored images to obtain reference image features corresponding to each image feature in the to-be-restored images;

[0022] Based on colors of the reference image features corresponding to the image features, perform color restoration on each image feature respectively to obtain the target images.

[0023] In an optional implementation, after the feature matching is performed on the reference image features of the split-shot reference image and the image features of the to-be-restored images, and color restoration is performed on the to-be-restored images based on a matching result to obtain target images, the method further includes:

[0024] The color of the target image is adjusted based on the target image and a color curve corresponding to a neighboring frame image of the target image, and the adjusted target image is taken as the target image.

[0025] According to another aspect of the embodiments of the present application, an image color restoration device based on deep learning and a split-screen reference image is provided, which comprises:

[0026] An image classification module is configured to acquire multiple frames of images in a video, and classify the multiple frames of images based on scene information of the multiple frames of images to obtain multiple sets of image collections.

[0027] A split-screen reference image determination module is configured to determine split-screen reference images corresponding to the multiple sets of image collections respectively.

[0028] An image feature determination module is configured to extract features of the multiple frames of to-be-restored images in the image collections and the split-screen reference images respectively to obtain associated image features of the multiple frames of to-be-restored images and reference image features of the split-screen reference images.

[0029] A color restoration module is configured to perform feature matching between the reference image features of the split-screen reference images and the associated image features of the to-be-restored images, and perform color restoration on the to-be-restored images based on the matching results to obtain target images.

[0030] According to another aspect of the embodiments of the present application, a computer device is provided, which comprises a processor, a memory, a communication interface and a communication bus, and the processor, the memory and the communication interface complete communication with each other through the communication bus.

[0031] The memory is configured to store at least one executable instruction, and the executable instruction causes the processor to perform operations of any of the image color restoration methods.

[0032] According to still another aspect of the embodiments of the present application, a computer readable storage medium is provided, and the storage medium stores at least one executable instruction, and the executable instruction causes the image color restoration device based on deep learning and a split-screen reference image to perform operations of any of the image color restoration methods when the image color restoration device based on deep learning and a split-screen reference image runs.

[0033] The present application provides a computer program product comprising executable instructions for causing a computer to perform any of the image color restoration methods based on deep learning and a split-screen reference image.

[0034] The embodiment of the present application classifies multiple images in a video according to scenes, colors the image sets corresponding to different scenes respectively, and excludes the interference of images of different scenes, thereby improving the accuracy of image coloring and restoration; meanwhile, by setting a split shot reference image and establishing a time sequence association between image features of a to-be-restored image, the coloring and restoration of the image can consider the association between the front and rear images, thereby ensuring the authenticity of image coloring and restoration, so that the accuracy and authenticity of image coloring and restoration are ensured while the efficiency of image coloring and restoration is ensured.

[0035] The above description is only a summary of the technical solutions of the embodiments of the present application, in order to more clearly understand the technical means of the embodiments of the present application, and can be implemented according to the content of the specification, and in order to make the above and other purposes, features and advantages of the embodiments of the present application more obvious and easy to understand, the specific embodiments of the present application are described below. BRIEF DESCRIPTION OF DRAWINGS

[0036] The accompanying drawings are included to provide a further understanding of the application and are incorporated herein and constitute a part of the detailed description. It should be noted that in the accompanying drawings, the same or similar elements are referred to using the same reference numerals.

[0037] Figure 1 is a flow diagram of an image color restoration method based on deep learning and split shot reference image provided by the embodiments of the present application;

[0038] Figure 2 is a schematic diagram of color restoration of a to-be-restored image in an image color restoration method based on deep learning and split shot reference image provided by the embodiments of the present application;

[0039] Figure 3 is a structure diagram of an embodiment of an image color restoration device based on deep learning and split shot reference image provided by the embodiments of the present application;

[0040] Figure 4 is a structure diagram of an embodiment of a computer device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0041] Exemplary embodiments of the present application will be described more fully hereinafter with reference to the accompanying drawings. Although exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein.

[0042] The current coloring method for black and white video mainly includes manual coloring, user interaction coloring and full-automatic coloring. In the manual coloring method, a large amount of time is needed to ensure the coloring effect of the video, which is low in efficiency and high in cost. In the user interaction coloring method, the quality and efficiency are improved compared with the manual coloring method, and the user guidance is reduced in the actual coloring process, but the coloring effect mainly depends on the semantic similarity between the reference sample and the input grayscale image. When the similarity is high, the coloring quality of the video is high, but when the similarity is low, color overflow and color artifacts are prone to occur, and the coloring result is single and not suitable for differentiated batch coloring. In the full-automatic coloring method, the missing color channel of the image is fitted based on big data and deep neural network to realize color prediction. Although the efficiency is high, there are problems such as color overflow, dullness and deviation, and the predicted color does not match the picture. At the same time, the coloring result is often diverse, and the same black and white video will have multiple reasonable coloring results, and is easily affected by the mainstream color, resulting in a coloring effect that is not real and accurate.

[0043] Based on this, the embodiment of the present application provides an image color restoration method based on deep learning and split shot reference image, which comprises: acquiring multiple frames of images in a video, and classifying the multiple frames of images based on scene information of the multiple frames of images to obtain multiple groups of image sets; determining split shot reference images corresponding to the multiple groups of image sets respectively; extracting features of the multiple frames of images to be restored in the image sets and the split shot reference images respectively to obtain associated image features of the multiple frames of images to be restored and reference image features of the split shot reference images; the associated image features of the multiple frames of images to be restored are time sequence associated, and the time sequence association corresponds to the order of the multiple frames of images to be restored in the image set; the reference image features of the split shot reference images are matched with the associated image features of the images to be restored, and the images to be restored are color restored based on the matching result to obtain target images. In this way, by classifying the multiple frames of images in the video according to the scene, the images in the image sets corresponding to different scenes are colored respectively, the interference of the images in different scenes is excluded, and the accuracy of image coloring and restoration is improved. At the same time, by setting the split shot reference images and establishing the time sequence association between the image features of the images to be restored, the coloring and restoration of the images can consider the correlation between the front and rear frames of images, so as to ensure the authenticity of the image coloring and restoration. Therefore, the efficiency of the image coloring and restoration is ensured, and the accuracy and authenticity of the image coloring and restoration are ensured.

[0044] The following introduces a specific embodiment of an image color restoration method based on deep learning and split shot reference image, Figure 1is a flowchart of an image color restoration method provided by an embodiment of the present application. The present specification provides method operation steps as embodiments or flowcharts, but more or fewer operation steps can be included based on conventional or non-creative labor. The order of steps listed in the embodiments is only one of the many step execution orders, and does not represent the only execution order. In actual system or server product execution, the method order shown in the embodiments or the drawings can be executed in sequence or in parallel (for example, in a parallel processor or multi-threaded processing environment). Specifically as shown in Figure 1 The method can include:

[0045] Step 110: Obtain multiple frames of images in a video, and classify the multiple frames of images based on scene information of the multiple frames of images to obtain multiple sets of image collections.

[0046] In the embodiments of the present application, multiple frames of images in a video that need to be color restored are obtained. In order to avoid low color restoration accuracy caused by low image similarity when color restoration is simultaneously performed on images of different scenes or different environments, the multiple frames of images are classified based on scene information of the images, and multiple sets of image collections are obtained according to the scene information classification, thereby realizing shot processing of the images contained in the video.

[0047] Specifically, first, scene information of the multiple frames of images is determined based on video cut information in the multiple frames of images; the video cut information is the position of a video cut shot in the video, and the video cut shot can realize the conversion of the view angle and the lens. Based on the video cut information, the multiple frames of images are classified, and images between two adjacent video cut shots are classified as images with the same scene information; then, the images with the same scene information are classified into one set of image collections, thereby realizing scene-based classification and integration of the multiple frames of images to obtain multiple sets of image collections.

[0048] In an optional embodiment, after the multiple sets of image groups are classified, each image in the multiple sets of image groups can be pre-processed and repaired. Specifically, the images in the multiple sets of image groups are pre-processed respectively, and non-target pixels in the images are removed, wherein the non-target pixels are pixels in the images that do not belong to the base picture of the images, that is, corresponding pixel points of stains, dirty spots, noise, jitter, and division in the images caused by some external reasons. The pre-processing of the images can be processed according to a preset repair algorithm, and after the repair algorithm is processed, it can be reported to a technician for further manual repair. Based on the brightness histogram, the brightness of the images in the multiple sets of image groups is adjusted respectively, so that the brightness value range of the images meets the standard, and the color value can also be adjusted according to the color vector diagram and the color histogram. Finally, considering that the dynamic range of the video is relatively flat and narrow, the details of the bright and dark parts of the picture are lost seriously, therefore, the contrast and saturation of the images in the multiple sets of image groups are adjusted respectively to enhance the detail information in the images, thereby enhancing the spatial sense and level sense of the picture.

[0049] Step 120: respectively determining the corresponding shot reference images of the multiple sets of image groups.

[0050] In the embodiment of the application, the corresponding shot reference images are selected for the multiple sets of image groups, and the images in each set of image groups are color restored with reference to the corresponding shot reference images in subsequent color restoration, that is, a reference image is selected for each set of image groups obtained by shot, and each shot is color restored respectively. The shot reference image is color restored after being selected, and the method of color restoring the shot reference image is not limited here, which can be manually color fixed by artificial or colored based on deep learning.

[0051] Step 130: respectively extracting features from the multiple frames of to-be-restored images in the image groups and the shot reference images to obtain the associated image features of the multiple frames of to-be-restored images and the reference image features of the shot reference images.

[0052] In the embodiment of the application, the associated image features of the multiple frames of to-be-restored images have a time sequence association, which corresponds to the order of the multiple frames of to-be-restored images in the image groups, so that the dynamic change of the video, that is, the dynamic change between the to-be-restored images, can be considered when the to-be-restored images are color restored, thereby ensuring the continuity of the color in the video after color restoration.

[0053] In an optional embodiment, Figure 2 is a schematic diagram of color restoration of to-be-restored images in an image color restoration method based on deep learning and shot reference images provided by the embodiment of the application, as Figure 2As shown, on one hand, the split-screen reference image and the plurality of frames of to-be-restored images are input into a first neural network to obtain reference image features and image features of the plurality of frames of to-be-restored images, wherein the first neural network can be a deep convolutional neural network and is mainly used for feature recognition and extraction; on the other hand, the plurality of frames of to-be-restored images are sequentially input into a second neural network according to the order of the to-be-restored images in the image set to obtain the time sequence correlation of the plurality of frames of to-be-restored images, wherein the second neural network can be a recurrent neural network and is used to analyze and obtain the dynamic changes between the to-be-restored images in each image set to obtain the time sequence correlation between the to-be-restored images. Finally, the outputs of the first neural network and the second neural network are combined, the time sequence correlation between the image features of the plurality of frames of to-be-restored images is established based on the time sequence correlation of the plurality of frames of to-be-restored images, the correlated image features are obtained, the semantic features of the to-be-restored images are constructed, and the time sequence features between the images can be considered in restoring the color of the images, so that the authenticity and accuracy of color restoration are ensured.

[0054] Step 140: performing feature matching on the reference image features of the split-screen reference image and the correlated image features of the to-be-restored images, and performing color restoration on the to-be-restored images based on the matching result to obtain a target image.

[0055] In the embodiment of the present application, after the correlated image features are constructed, feature matching is performed on the reference image features and the correlated image features, the reference image features matched with the correlated image features are determined, and the color of the correlated image features is restored based on the matched reference image features, so that the color of the to-be-restored images is restored to obtain a target image after color restoration.

[0056] Specifically, the image features of the split-screen reference image and the image features of the to-be-restored images are matched through the feature matching subnetwork to obtain the reference image features corresponding to each image feature in the to-be-restored images. Then, the color of the reference image features corresponding to the image features is extracted as a coloring sample, and each pixel in the correlated image features is matched in color through the coloring subnetwork, so that each pixel is colored, and finally each image feature is color restored to obtain a target image.

[0057] In an optional implementation, after the to-be-restored images are color restored to obtain a target image, the color of the target image is adjusted based on the target image and the color curve corresponding to the adjacent frame image of the target image, the adjusted target image is taken as the target image, so that the color consistency of the front and rear images and the rationality of the color in multi-scene conversion are ensured, and the authenticity of color restoration is improved.

[0058] The image color restoration method based on deep learning and split-screen reference images provided by the embodiment of the present application comprises the following steps: obtaining multiple frames of images in a video, classifying the multiple frames of images based on scene information of the multiple frames of images to obtain multiple sets of image collections, determining split-screen reference images corresponding to the multiple sets of image collections respectively, extracting features of the multiple frames of images to be restored in the image collections and the split-screen reference images respectively to obtain associated image features of the multiple frames of images to be restored and reference image features of the split-screen reference images, the associated image features of the multiple frames of images to be restored being time-sequentially associated, the time-sequential association corresponding to an order of the multiple frames of images to be restored in the image collections, performing feature matching on the reference image features of the split-screen reference images and the associated image features of the images to be restored, and performing color restoration on the images to be restored based on a matching result to obtain target images.

[0059] The embodiment of the present application further provides an image color restoration device based on deep learning and split-screen reference images, Figure 3 is a structural schematic diagram of an embodiment of the image color restoration device provided by the embodiment of the present application, as Figure 3 shown, the device 300 comprises:

[0060] An image classification module 310 is configured to obtain multiple frames of images in a video, and classify the multiple frames of images based on scene information of the multiple frames of images to obtain multiple sets of image collections.

[0061] A split-screen reference image determination module 320 is configured to determine split-screen reference images corresponding to the multiple sets of image collections respectively.

[0062] An image feature determination module 330 is configured to extract features of multiple frames of images to be restored in the image collections and the split-screen reference images respectively to obtain associated image features of the multiple frames of images to be restored and reference image features of the split-screen reference images, the associated image features of the multiple frames of images to be restored being time-sequentially associated, the time-sequential association corresponding to an order of the multiple frames of images to be restored in the image collections.

[0063] A color restoration module 340 is configured to perform feature matching on the reference image features of the split-screen reference images and the associated image features of the images to be restored, and perform color restoration on the images to be restored based on a matching result to obtain target images.

[0064] In an optional implementation, the image classification module 310 comprises:

[0065] a scene information determination unit configured to determine scene information of the multiple frames of images based on the video cut information in the multiple frames of images;

[0066] an image classification unit configured to classify images with the same scene information into a set of image groups to obtain multiple sets of image groups.

[0067] In an optional implementation, the apparatus further comprises:

[0068] a preprocessing module configured to respectively preprocess images in the multiple sets of image groups to remove non-target pixels of the images in the sets of image groups; the non-target pixels are pixels in the images that do not belong to a base frame of the images;

[0069] a brightness adjustment module configured to respectively adjust brightness of the images in the multiple sets of image groups based on a brightness histogram;

[0070] a contrast adjustment module configured to respectively adjust contrast and saturation of the images in the multiple sets of image groups.

[0071] In an optional implementation, the image feature determination module 330 comprises:

[0072] an image feature determination unit configured to input the shot reference images and the multiple frames of images to be restored into a first neural network to obtain reference image features and image features of the multiple frames of images to be restored;

[0073] a time sequence association determination unit configured to input the multiple frames of images to be restored into a second neural network in sequence according to an order of the images to be restored in the sets of images to obtain time sequence associations of the multiple frames of images to be restored;

[0074] an associated image feature determination unit configured to establish time sequence associations between the image features of the multiple frames of images to be restored based on the time sequence associations of the multiple frames of images to be restored to obtain associated image features.

[0075] In an optional implementation, the color restoration module 340 comprises:

[0076] a feature matching unit configured to match the image features of the shot reference images with the image features of the images to be restored to obtain reference image features corresponding to each of the image features in the images to be restored;

[0077] a color restoration unit configured to respectively perform color restoration on each of the image features based on colors of the reference image features corresponding to the image features to obtain target images.

[0078] In an optional implementation, the apparatus further comprises:

[0079] a color adjustment module configured to adjust a color of the target image based on the color curve corresponding to the target image and a neighboring frame image of the target image, and take the adjusted target image as the target image.

[0080] The device and method embodiments in the present application are based on the same inventive concept.

[0081] Figure 4 The present application provides a computer device, and a structure diagram of an embodiment of the computer device is shown. The specific embodiments of the present application do not limit the specific implementation of the computer device.

[0082] As shown in Figure 4 The computer device can include a processor 402, a communications interface 404, a memory 406, and a communications bus 408.

[0083] The processor 402, the communications interface 404, and the memory 406 can communicate with each other through the communications bus 408. The communications interface 404 is configured to communicate with network elements such as clients or other servers. The processor 402 is configured to execute the program 410, and specifically can execute the related steps in the above-mentioned image color restoration method embodiments.

[0084] Specifically, the program 410 can include program codes including computer executable instructions.

[0085] The processor 402 can be a central processing unit CPU, or an application specific integrated circuit ASIC, or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the computer device can be the same type of processor, such as one or more CPUs; or can be different types of processors, such as one or more CPUs and one or more ASICs.

[0086] The memory 406 is configured to store the program 410. The memory 406 can include a high-speed RAM memory, and can also include a non-volatile memory such as at least one disk memory.

[0087] The program 410 can be specifically invoked by the processor 402 to enable the computer device to execute the related steps in the above-mentioned image color restoration method embodiments.

[0088] Those skilled in the art can understand that Figure 4The illustrated structure is merely schematic and does not limit the structure of the above-mentioned device. For example, the computer device can further include more or less components than those shown, or have a different configuration of components than that shown. Figure 4 The illustrated structure is merely schematic and does not limit the structure of the above-mentioned device. For example, the computer device can further include more or less components than those shown, or have a different configuration of components than that shown. Figure 4 The illustrated structure is merely schematic and does not limit the structure of the above-mentioned device. For example, the computer device can further include more or less components than those shown, or have a different configuration of components than that shown.

[0089] The computer readable storage medium stores at least one executable instruction, which, when executed on the image color restoration device based on deep learning and shot reference image, causes the image color restoration device based on deep learning and shot reference image to perform the image color restoration method in any of the above method embodiments.

[0090] Part of the present application can be applied as a computer program product, for example, executable instructions, when executed by a computer, through the operation of the computer, the method and / or technical solutions according to the present application can be called or provided. Those skilled in the art should understand that the executable instructions in the computer readable medium include but are not limited to source files, executable files, installation package files, etc. Correspondingly, the executable instructions executed by the computer include but are not limited to: the computer directly executes the instructions, or the computer executes the corresponding compiled program after compiling the instructions, or the computer reads and executes the instructions, or the computer reads and installs the instructions and then executes the corresponding installed program. Here, the computer readable medium can be any available computer readable storage medium or communication medium accessible to the computer.

[0091] The algorithms or displays provided herein are not inherently related to any particular computer, virtual system, or other apparatus. Furthermore, the embodiments of the present application are not described with reference to any particular programming language.

[0092] In the specification provided herein, a large number of specific details are described. However, it can be understood that the embodiments of the present application can be practiced without these specific details. Similarly, in order to simplify the present application and help understand one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present application, various features of the embodiments of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. Among them, the claims following the detailed description are hereby expressly incorporated into the detailed description, wherein each claim itself is a separate embodiment of the present application.

[0093] It will be appreciated by those skilled in the art that modules in the apparatuses in the embodiments can be adapted and placed in one or more apparatuses other than that of the embodiments. Modules or units or components in the embodiments can be combined into one module or unit or component and furthermore can be divided into multiple sub-modules or sub-units or sub-components. Except that at least some of such features and / or processes or units are mutually exclusive.

[0094] It should be noted that the above-mentioned embodiments illustrate rather than limit the application, and that those skilled in the art will be able to design many alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word 'comprising' does not exclude the presence of elements or steps other than those listed in a claim. The word 'a' or 'an' preceding an element does not exclude the presence of a plurality of such elements. The application can be implemented by means of both hardware and software, and any combination thereof. In a unit claim, any reference to 'composition' should not be construed as a limitation unless the composition is a product of manufacturing. The use of the word 'about' in relation to a numerical value preferably means ± 10 % of the value. The word 'first','second', 'third', etc. does not imply any order. The use of the terms 'first' and'second' are merely intended to differentiate between two distinct units. It is to be understood that the steps in the above-described embodiments can not be performed in the order described, unless specifically stated otherwise.

Claims

1. An image color restoration method based on deep learning and storyboard reference images, characterized in that, The method includes: The process involves acquiring multiple frames of images from a video and classifying them based on scene information to obtain multiple image sets. This classification includes: determining scene information based on video cut-out information within the multiple frames; and grouping images with the same scene information into a single image set to obtain multiple image sets. The video cut-out information refers to the position of the video cut-out shot within the video, which is used to achieve perspective and camera movement. Each of the aforementioned image sets is assigned a set of storyboard reference images; the storyboard reference images are obtained by manually colorizing or colorizing reference images based on deep learning. Feature extraction is performed on multiple frames of images to be restored and the storyboard reference images in the image set to obtain associated image features of the multiple frames of images to be restored and reference image features of the storyboard reference images. Temporal correlations exist between the associated image features of the multiple frames of images to be restored, and these temporal correlations correspond to the order of the multiple frames of images to be restored in the image set. The step of extracting features from multiple frames of images to be restored and the storyboard reference images in the image set to obtain associated image features of the multiple frames of images to be restored and reference image features of the storyboard reference images includes: inputting the storyboard reference images and multiple frames of images to be restored into a first neural network to obtain the reference image features and the image features of the multiple frames of images to be restored, wherein the first neural network is a deep convolutional neural network; inputting the multiple frames of images to be restored sequentially into a second neural network according to the order of the images to be restored in the image set to obtain the temporal correlations of the multiple frames of images to be restored, wherein the second neural network is a recurrent neural network; and establishing temporal correlations between the image features of the multiple frames of images to be restored based on these temporal correlations to obtain the associated image features. The reference image features of the storyboard reference image are matched with the associated image features of the image to be restored, and the color of the image to be restored is restored based on the matching results to obtain the target image.

2. The method according to claim 1, characterized in that, After classifying the multi-frame images based on scene information to obtain multiple image sets, the method further includes: The images in the multiple sets of images are preprocessed to remove non-target pixels from the images in the image sets; the non-target pixels are pixels in the images that do not belong to the basic image frame. Based on the brightness histogram, the brightness of the images in the multiple sets of images is adjusted respectively; The contrast and saturation of the images in the multiple sets of images are adjusted respectively.

3. The method according to claim 1, characterized in that, The step of performing feature matching between the reference image features of the storyboard reference image and the associated image features of the image to be restored, and performing color restoration on the image to be restored based on the matching result to obtain the target image, includes: The reference image features of the storyboard reference image are matched with the image features of the image to be restored to obtain the reference image features corresponding to each image feature in the image to be restored. Based on the color of the reference image features corresponding to the image features, color restoration is performed on each of the image features to obtain the target image.

4. The method according to claim 1, characterized in that, After performing feature matching between the reference image features of the storyboard reference image and the image features of the image to be restored, and performing color restoration on the image to be restored based on the matching result to obtain the target image, the method further includes: Based on the target image and the color curves corresponding to its adjacent frames, the color of the target image is adjusted, and the adjusted target image is used as the target image.

5. An image color restoration device based on deep learning and storyboard reference images, characterized in that, The device includes: An image classification module is used to acquire multiple frames of images from a video and classify the multiple frames of images based on scene information to obtain multiple image sets. The classification of the multiple frames of images based on scene information to obtain multiple image sets includes: determining the scene information of the multiple frames of images based on video cut-out information; classifying images with the same scene information into one image set to obtain multiple image sets; the video cut-out information refers to the position of the video cut-out shot in the video, and the video cut-out shot is used to realize the switching of viewpoint and camera angle. The storyboard reference image determination module is used to determine storyboard reference images corresponding to multiple sets of images; the storyboard reference images are obtained by manually colorizing reference images or by colorizing them based on deep learning. The image feature determination module is used to extract features from multiple frames of images to be restored and the storyboard reference images in the image set, respectively, to obtain associated image features of the multiple frames of images to be restored and reference image features of the storyboard reference images; there is a temporal correlation among the associated image features of the multiple frames of images to be restored, and the temporal correlation corresponds to the order of the multiple frames of images to be restored in the image set; the step of extracting features from multiple frames of images to be restored and the storyboard reference images in the image set to obtain associated image features of the multiple frames of images to be restored and reference image features of the storyboard reference images is described. The method includes: inputting the storyboard reference image and multiple frames of the images to be restored into a first neural network to obtain the reference image features and the image features of the multiple frames of the images to be restored, wherein the first neural network is a deep convolutional neural network; inputting the multiple frames of the images to be restored into a second neural network in the order of the images to be restored in the image set to obtain the temporal association of the multiple frames of the images to be restored, wherein the second neural network is a recurrent neural network; and establishing a temporal association between the image features of the multiple frames of the images to be restored based on the temporal association of the multiple frames of the images to be restored to obtain the associated image features. The color restoration module is used to perform feature matching between the reference image features of the storyboard reference image and the associated image features of the image to be restored, and to perform color restoration on the image to be restored based on the matching result to obtain the target image.

6. A computer device, characterized in that, include: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction that causes the processor to perform the operation of the image color restoration method based on deep learning and storyboard reference images as described in any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The storage medium stores at least one executable instruction, which, when executed on the image color restoration device / device, causes the image color restoration device / device to perform the operation of the image color restoration method based on deep learning and storyboard reference images as described in any one of claims 1-4.

8. A computer program product, characterized in that, It includes executable instructions for causing a computer to perform the image color restoration method based on deep learning and storyboard reference images as described in any one of claims 1 to 4.

Images