Movie image object recognition and automatic color restoration system based on deep learning

Abstract

The present invention discloses a deep learning-based movie image object recognition and automatic color restoration system, which relates to the field of image recognition technology and includes: an image input module for receiving movie image frames to be restored; an object recognition module for identifying specific objects appearing in the image frames based on a deep learning algorithm; a color mapping module for color restoration of the identified objects; a restoration processing module for restoring the identified and colored objects based on an automated algorithm, restoring their color and details; and an image output module for outputting the restored movie image frames. The present invention proposes automatic recognition and color restoration of objects in images, avoiding the tedious and inefficient traditional manual restoration process. Through a deep learning model, the system can intelligently restore the color and details of objects based on their features in the image. Through feature extraction and object detection using a multi-layer convolutional neural network, the system can efficiently identify specific objects in images.

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology, and in particular to a movie image object recognition and automatic coloring and restoration system based on deep learning. Background Art

[0002] With the rapid development of deep learning technology, the field of image processing has made significant progress in many applications, especially in image restoration, object recognition, and image colorization. Film images, as a special type of image, often contain complex scenes and object details. Therefore, traditional image processing methods often fail to meet the high-precision and high-quality requirements for film image restoration and colorization. During the film image restoration process, many classic film images may become damaged or faded over time, resulting in a loss of image detail and color, especially in black-and-white or older films. For these damaged images, traditional manual restoration methods are not only inefficient but also have poor restoration effects and cannot automatically identify and restore specific objects and colors in the film.

[0003] At present, many deep learning-based image restoration technologies focus on edge restoration, texture filling, color mapping and detail restoration. However, there are still some challenges in tasks such as automatic object recognition, color mapping and color restoration of movie images, such as how to accurately identify specific objects in the image, how to adjust the color in combination with contextual information, how to restore the details of the object and lighting effects, etc. Therefore, the present invention proposes a movie image object recognition and automatic color restoration system based on deep learning. The system uses the advantages of deep learning to automatically identify specific objects in movie images and repair and color them, which can achieve efficient and accurate movie image restoration, especially the restoration of old movie images. Summary of the Invention

[0004] In order to solve the above technical problems, a movie image object recognition and automatic color restoration system based on deep learning is provided. This technical solution solves the above problems.

[0005] In order to achieve the above objects, the technical solution adopted by the present invention is:

[0006] A deep learning-based movie image object recognition and automatic color restoration system, including:

[0007] Image input module: The image input module is used to receive the movie image frame to be restored;

[0008] Object recognition module: The object recognition module is electrically connected to the image input module and is used to identify specific objects appearing in image frames based on a deep learning algorithm. The specific objects are objects with specific structures and characteristics that appear repeatedly in the movie;

[0009] Color mapping module: The color mapping module is electrically connected to the object recognition module and is used to perform color restoration on the recognized object through the model based on the known object color information;

[0010] Repair processing module: The repair processing module is electrically connected to the color mapping module, and is used to repair the identified and colored objects based on an automated algorithm to restore their colors and details;

[0011] Image output module: The image output module is electrically connected to the restoration processing module, and is used to output the restored movie image frames.

[0012] Preferably, the image input module specifically includes:

[0013] Image acquisition unit: receives the movie image frames to be restored. The image frames are extracted from the video files and input through the real-time video stream;

[0014] Preprocessing unit: performs preliminary preprocessing on the received movie images, including denoising, grayscale and contrast adjustment;

[0015] Frame processing unit: decomposes the movie video into independent image frames, and each frame is analyzed and repaired as an independent processing unit;

[0016] Size normalization unit: performs size normalization on the input image frames to ensure that all input images have the same resolution and size;

[0017] Data format conversion unit: According to the requirements of subsequent modules, the input image is converted into a matrix data format and the received image frames are temporarily stored.

[0018] Preferably, the object recognition module includes:

[0019] Feature extraction unit: Based on the deep learning object recognition model, multiple convolution layers are used to extract features. The feature extraction process mainly includes convolution operations and pooling operations to extract local features of the image, including edges, textures, and colors.

[0020] Category output unit: Based on recurring objects with specific structures and features, a convolutional neural network is used to train an object recognition model. Through multiple layers of convolution, activation, pooling, and fully connected layers, the spatial features in the image are learned and the object category in the image is output.

[0021] Object detection and classification unit: detects specific objects in image frames, locates objects, and classifies objects;

[0022] Post-processing unit: After the object recognition module completes object detection, the non-maximum suppression method is used to eliminate duplicate detection frames.

[0023] Preferably, the feature extraction unit specifically includes:

[0024] Based on the convolutional neural network, feature extraction is performed through multiple convolutional layers. The convolutional neural network uses the convolution kernel to perform convolution operations on the image to extract local features of the image, including edges, textures, and colors. The convolution operation formula is:

[0025]

[0026] Where I is the input image, K is the convolution kernel, (m, n) is the coordinate of the convolution kernel, K(m, n) is the convolution kernel, (x, y) is the position coordinate in the output feature map, and (I·K)(x, y) represents the pixel value at position (x, y) in the output image after the convolution operation.

[0027] Based on the pooling operation, the maximum pooling method is used to reduce the size of the feature map, reduce the amount of calculation, and retain important feature information. The formula of the maximum pooling operation is:

[0028]

[0029] Among them, P(x, y) represents the pixel value at position (x, y) in the output image after the pooling operation, I(x+m, y+n) represents the pixel value at position (x+m, y+n) in the input image, and max m，n Indicates taking the maximum value within the pooling window during the pooling operation.

[0030] Preferably, the object detection and classification unit specifically includes:

[0031] Detect specific objects in the image frame and locate the objects through the output of the model. The position of the objects in the image is determined based on the bounding box coordinates output by the model.

[0032] Perform object classification and identify the object categories in the image. The model outputs a probability distribution, which is the probability that the object in the image belongs to each category, and selects the category with the highest probability as the predicted category of the object.

[0033] Preferably, the post-processing unit specifically includes:

[0034] After the object recognition module completes object detection, it uses non-maximum suppression post-processing technology to eliminate duplicate detection frames and retain the most likely detection frames to prevent the same object from being recognized multiple times. The algorithm steps of non-maximum suppression are as follows:

[0035] Sort bounding boxes by classification confidence;

[0036] Select the bounding box with the highest confidence as the reference;

[0037] Remove the boxes that have a large overlap with the reference box. The formula for calculating the intersection of two bounding boxes is:

[0038]

[0039] Where A represents the predicted area, B represents the true area, |A∩B| represents the area of ​​the intersection of A and B, |A∪B| represents the area of ​​the union of A and B, and IOU is the evaluation indicator used to measure the two areas.

[0040] Preferably, the color mapping module specifically includes:

[0041] Output acquisition unit of the object recognition module: The object recognition module performs object recognition based on the input data, identifying each object in the image as well as their category and location;

[0042] Object category information: Recognized objects are classified into different categories, including "apple", "car" and "chair";

[0043] Position and shape information: The specific position and shape information of the object, including the coordinates of the bounding box and a more precise segmentation mask, will also be passed to the color mapping module;

[0044] Color information matching unit: Based on the category and context information of the object, the color mapping module matches the typical color of the object in the category from the database;

[0045] Object region coloring unit: Extracts the region of the object in the image based on the results of object recognition and defines it by bounding box and segmentation mask.

[0046] Preferably, the object area coloring unit specifically includes:

[0047] Color mapping and restoration: Apply selected color information to the object area for coloring. The color of the object in the image has been partially lost, so restoration is performed to restore the typical color of the object. The restoration process includes:

[0048] Color filling: If the color of the identified area is lost or inaccurate, the color is restored through the filling algorithm;

[0049] Model-based color adjustment: If there are changes in lighting, shadows, etc. on the surface of an object, the color mapping module needs to take these factors into account and make appropriate color adjustments;

[0050] Color mapping optimization and fusion: Perform color smoothing, use interpolation to make a smooth transition between the object color and the background color, and perform color correction based on the object's spatial position and lighting information.

[0051] Preferably, the repair processing module specifically includes:

[0052] Object recognition and coloring information acquisition unit: receives the colored object information from the color mapping module and identifies the object's area, boundaries, and missing texture details;

[0053] Image restoration region analysis unit: Determines the edge position of an object based on an edge detection algorithm, identifies the object's outline, checks for missing textures and details, and supplements them with filling and interpolation algorithms;

[0054] Detail recovery and texture restoration unit: Restores these details based on the texture restoration algorithm, fills the image based on the surrounding pixels, and is used to restore texture and color details, enhance areas with missing or blurred textures, and increase the clarity of details;

[0055] Color restoration and adjustment unit: performs local color adjustments based on the object's ambient lighting, shadow areas, and other contextual information, adjusting the global and local color balance of the image;

[0056] Detail enhancement and lighting restoration unit: performs lighting restoration based on the lighting conditions on the object surface, restores natural lighting effects, and adjusts the brightness and contrast of the image through HDR technology.

[0057] Preferably, the image restoration region analysis unit specifically includes:

[0058] The edge position of the object is determined based on the edge detection algorithm, and the object contour is identified. The edge detection algorithm formula is:

[0059]

[0060] In the formula, G represents the gradient size of a certain position in the image, G x Represents the horizontal gradient of a certain position in the image, G y It represents the vertical gradient of a certain position in the image, θ represents the gradient direction of a certain position in the image, and atan2 is a mathematical function.

[0061] Compared with the prior art, the present invention has the following beneficial effects:

[0062] The present invention proposes a method for automatically identifying objects in images and coloring and repairing them, avoiding the tedious and inefficient traditional manual repair process. Through a deep learning model, the system can intelligently restore the color and details of the object based on the object features in the image. Through feature extraction and object detection through a multi-layer convolutional neural network, the system can efficiently identify specific objects in the image, especially those that appear repeatedly in movies and have specific structures and features. Based on these recognition results, the system can accurately locate and classify the objects, providing accurate input for subsequent repair and coloring. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 It is a system framework diagram of the present invention;

[0064] Figure 2 This is a diagram of the internal system framework of the image input module of the present invention;

[0065] Figure 3 This is a diagram of the internal system framework of the object recognition module in the present invention;

[0066] Figure 4 This is a diagram of the internal system framework of the color mapping module in the present invention;

[0067] Figure 5 This is the internal system framework diagram of the repair processing module in the present invention. DETAILED DESCRIPTION

[0068] The following description is intended to disclose the present invention so that those skilled in the art can implement the present invention. The preferred embodiments described below are merely examples, and those skilled in the art may conceive of other obvious variations.

[0069] Reference Figure 1 As shown in the figure, the deep learning-based movie image object recognition and automatic color restoration system includes:

[0070] Step 1:

[0071] Image acquisition unit: receives the movie image frames to be restored. The image frames are extracted from the video files and input through the real-time video stream;

[0072] Preprocessing unit: performs preliminary preprocessing on the received movie images, including denoising, grayscale and contrast adjustment;

[0073] Frame processing unit: decomposes the movie video into independent image frames, and each frame is analyzed and repaired as an independent processing unit;

[0074] Size normalization unit: performs size normalization on the input image frames to ensure that all input images have the same resolution and size;

[0075] Data format conversion unit: According to the requirements of subsequent modules, the input image is converted into a matrix data format and the received image frames are temporarily stored.

[0076] Step 2:

[0077] Feature extraction unit: Based on the deep learning object recognition model, multiple convolution layers are used to extract features. The feature extraction process mainly includes convolution operations and pooling operations to extract local features of the image, including edges, textures, and colors.

[0078] Based on the convolutional neural network, feature extraction is performed through multiple convolutional layers. The convolutional neural network uses the convolution kernel to perform convolution operations on the image to extract local features of the image, including edges, textures, and colors. The convolution operation formula is:

[0079]

[0080] Where I is the input image, K is the convolution kernel, (m, n) is the coordinate of the convolution kernel, K(m, n) is the convolution kernel, (x, y) is the position coordinate in the output feature map, and (I·K)(x, y) represents the pixel value at position (x, y) in the output image after the convolution operation.

[0081] Based on the pooling operation, the maximum pooling method is used to reduce the size of the feature map, reduce the amount of calculation, and retain important feature information. The formula of the maximum pooling operation is:

[0082]

[0083] Among them, P(x, y) represents the pixel value at position (x, y) in the output image after the pooling operation, I(x+m, y+n) represents the pixel value at position (x+m, y+n) in the input image, and max m，n Indicates taking the maximum value within the pooling window during the pooling operation.

[0084] Category output unit: Based on recurring objects with specific structures and features, a convolutional neural network is used to train an object recognition model. Through multiple layers of convolution, activation, pooling, and fully connected layers, the spatial features in the image are learned and the object category in the image is output.

[0085] Object detection and classification unit: detects specific objects in image frames, locates objects, and classifies objects;

[0086] Detect specific objects in the image frame and locate the objects through the output of the model. The position of the objects in the image is determined based on the bounding box coordinates output by the model.

[0087] Perform object classification and identify the object categories in the image. The model outputs a probability distribution, which is the probability that the object in the image belongs to each category, and selects the category with the highest probability as the predicted category of the object.

[0088] Post-processing unit: After the object recognition module completes object detection, it uses the non-maximum suppression method to eliminate duplicate detection frames;

[0089] After the object recognition module completes object detection, it uses non-maximum suppression post-processing technology to eliminate duplicate detection frames and retain the most likely detection frames to prevent the same object from being recognized multiple times. The algorithm steps of non-maximum suppression are as follows:

[0090] Sort bounding boxes by classification confidence;

[0091] Select the bounding box with the highest confidence as the reference;

[0092] Remove the boxes that have a large overlap with the reference box. The formula for calculating the intersection of two bounding boxes is:

[0093]

[0094] Where A represents the predicted area, B represents the true area, |A∩B| represents the area of ​​the intersection of A and B, |A∪B| represents the area of ​​the union of A and B, and IOU is the evaluation indicator used to measure the two areas.

[0095] Step 3:

[0096] Output acquisition unit of the object recognition module: The object recognition module performs object recognition based on the input data, identifying each object in the image as well as their category and location;

[0097] Object category information: Recognized objects are classified into different categories, including "apple", "car" and "chair";

[0098] Position and shape information: The specific position and shape information of the object, including the coordinates of the bounding box and a more precise segmentation mask, will also be passed to the color mapping module;

[0099] Color information matching unit: Based on the category and context information of the object, the color mapping module matches the typical color of the object in the category from the database;

[0100] Object region coloring unit: extracts the region of the object in the image based on the object recognition results, and defines it by bounding box and segmentation mask;

[0101] Color mapping and restoration: Apply selected color information to the object area for coloring. The color of the object in the image has been partially lost, so restoration is performed to restore the typical color of the object. The restoration process includes:

[0102] Color filling: If the color of the identified area is lost or inaccurate, the color is restored through the filling algorithm;

[0103] Model-based color adjustment: If there are changes in lighting, shadows, etc. on the surface of an object, the color mapping module needs to take these factors into account and make appropriate color adjustments;

[0104] Color mapping optimization and fusion: Perform color smoothing, use interpolation to make a smooth transition between the object color and the background color, and perform color correction based on the object's spatial position and lighting information.

[0105] Step 4:

[0106] Object recognition and coloring information acquisition unit: receives the colored object information from the color mapping module and identifies the object's area, boundaries, and missing texture details;

[0107] Image restoration region analysis unit: Determines the edge position of an object based on an edge detection algorithm, identifies the object's outline, checks for missing textures and details, and supplements them with filling and interpolation algorithms;

[0108] The edge position of the object is determined based on the edge detection algorithm, and the object contour is identified. The edge detection algorithm formula is:

[0109]

[0110] In the formula, G represents the gradient size of a certain position in the image, G x Represents the horizontal gradient of a certain position in the image, G y It represents the vertical gradient of a certain position in the image, θ represents the gradient direction of a certain position in the image, and atan2 is a mathematical function.

[0111] Detail recovery and texture restoration unit: Restores these details based on the texture restoration algorithm, fills the image based on the surrounding pixels, and is used to restore texture and color details, enhance areas with missing or blurred textures, and increase the clarity of details;

[0112] Color restoration and adjustment unit: performs local color adjustments based on the object's ambient lighting, shadow areas, and other contextual information, adjusting the global and local color balance of the image;

[0113] Detail enhancement and lighting restoration unit: performs lighting restoration based on the lighting conditions on the object surface, restores natural lighting effects, and adjusts the brightness and contrast of the image through HDR technology.

[0114] In summary, the advantages of the present invention are:

[0115] The system of the present invention can automatically identify objects in images and color them, avoiding the tedious and inefficient traditional manual restoration process. Through the deep learning model, the system can intelligently restore the color and details of objects based on the characteristics of the objects in the image.

[0116] By using a multi-layer convolutional neural network for feature extraction and object detection, the system can efficiently identify specific objects in images, especially those that appear repeatedly in movies and have specific structures and features. Based on these recognition results, the system can accurately locate and classify objects, providing accurate input for subsequent restoration and colorization.

[0117] The system automatically restores the typical color of the object through the color mapping module and adjusts the color according to the object's ambient lighting, shadows and other information to ensure a natural visual transition of the restored image. In addition, the restoration processing module can restore the texture and details of the object, using texture restoration and interpolation algorithms to supplement the details, making the object's surface more refined and the lighting effect more natural.

[0118] The use of deep learning-based image restoration technology makes the color and details of objects more realistic. Through color mapping optimization and fusion, color smoothing, and brightness and contrast restoration using HDR technology, the restored images are clearer and more delicate, greatly improving the visual quality of old movie images.

[0119] This system is not only able to adapt to the needs of restoring most objects in movie images, but can also handle complex situations such as different lighting conditions and shadow changes. It has strong adaptability and robustness. By introducing deep learning models, especially convolutional neural networks for feature extraction and object recognition, this system can ensure the accuracy of image restoration on the basis of high efficiency. The system can identify and repair key objects in the image, restore its realism and details, and make the movie image close to its original state in terms of visual effect.

[0120] The system of the present invention can be customized and expanded according to the needs of different movies or images. It can process various types of movie images and is not limited to black and white movies or old movies. It is also suitable for other image restoration and colorization tasks.

[0121] The above shows and describes the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions merely illustrate the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the appended claims and their equivalents.

Claims

1. A deep learning-based movie image object recognition and automatic color restoration system, characterized by: include: Image input module: The image input module is used to receive the movie image frame to be restored; Object recognition module: The object recognition module is electrically connected to the image input module and is used to identify specific objects appearing in image frames based on a deep learning algorithm. The specific objects are objects with specific structures and characteristics that appear repeatedly in the movie; Color mapping module: The color mapping module is electrically connected to the object recognition module and is used to perform color restoration on the recognized object through the model based on the known object color information; Repair processing module: The repair processing module is electrically connected to the color mapping module, and is used to repair the identified and colored objects based on an automated algorithm to restore their colors and details; Image output module: The image output module is electrically connected to the restoration processing module, and is used to output the restored movie image frame; The object recognition module includes: Feature extraction unit: Based on the deep learning object recognition model, multiple convolution layers are used to extract features; Category output unit: Based on recurring objects with specific structures and features, a convolutional neural network is used to train an object recognition model. Through multiple layers of convolution, activation, pooling, and fully connected layers, the spatial features in the image are learned and the object category in the image is output. Object detection and classification unit: detects specific objects in image frames, locates objects, and classifies objects; Post-processing unit: After the object recognition module completes object detection, it uses the non-maximum suppression method to eliminate duplicate detection frames; The feature extraction unit specifically includes: Based on the convolutional neural network, feature extraction is performed through multiple convolutional layers. The convolutional neural network uses the convolution kernel to perform convolution operations on the image to extract local features of the image, including edges, textures, and colors. The convolution operation formula is: ； Where, is the input image, is the convolution kernel, are the coordinates of the convolution kernel, is the convolution kernel, are the position coordinates in the output feature map, Indicates that the output image after the convolution operation is at position Pixel value of Based on the pooling operation, the maximum pooling method is used to reduce the size of the feature map, reduce the amount of calculation, and retain important feature information. The formula of the maximum pooling operation is: ； in, Indicates the position in the output image after the pooling operation The pixel value at Indicates that the input image is at position The pixel value at Indicates taking the maximum value within the pooling window during the pooling operation.

2. The deep learning-based movie image object recognition and automatic color restoration system according to claim 1, characterized in that: The image input module specifically includes: Image acquisition unit: receives the movie image frames to be restored. The image frames are extracted from the video files and input through the real-time video stream; Preprocessing unit: performs preliminary preprocessing on the received movie images, including denoising, grayscale and contrast adjustment; Frame processing unit: decomposes the movie video into independent image frames, and each frame is analyzed and repaired as an independent processing unit; Size normalization unit: performs size normalization on the input image frames to ensure that all input images have the same resolution and size; Data format conversion unit: According to the requirements of subsequent modules, the input image is converted into a matrix data format and the received image frames are temporarily stored.

3. The deep learning-based movie image object recognition and automatic color restoration system according to claim 1, characterized in that: The object detection and classification unit specifically includes: Detect specific objects in the image frame and locate the objects through the output of the model. The position of the objects in the image is determined based on the bounding box coordinates output by the model. Perform object classification and identify the object categories in the image. The model outputs a probability distribution, which is the probability that the object in the image belongs to each category, and selects the category with the highest probability as the predicted category of the object.

4. The deep learning-based movie image object recognition and automatic color restoration system according to claim 3, characterized in that: The post-processing unit specifically includes: After the object recognition module completes object detection, it uses non-maximum suppression post-processing technology to eliminate duplicate detection frames and retain the most likely detection frames to prevent the same object from being recognized multiple times. The formula for calculating the intersection of two bounding boxes is: ； Where, represents the prediction area, represents the real area, represents the area of ​​the intersection of A and B, represents the area of ​​the union of A and B, is the evaluation indicator used to measure the two areas.

5. The deep learning-based movie image object recognition and automatic color restoration system according to claim 4, characterized in that: The color mapping module specifically includes: Output acquisition unit of the object recognition module: The object recognition module performs object recognition based on the input data, identifying each object in the image as well as their category and location; Color information matching unit: Based on the category and context information of the object, the color mapping module matches the typical color of the object in the category from the database; Object region coloring unit: Extracts the region of the object in the image based on the results of object recognition and defines it by bounding box and segmentation mask.

6. The deep learning-based movie image object recognition and automatic color restoration system according to claim 5, characterized in that: The object area coloring unit specifically includes: Color mapping and restoration: Apply selected color information to the object area to perform coloring operations. If the color of the object in the image has been partially lost, restoration is performed to restore the typical color of the object; Color mapping optimization and fusion: Perform color smoothing, use interpolation to make a smooth transition between the object color and the background color, and perform color correction based on the object's spatial position and lighting information.

7. The deep learning-based movie image object recognition and automatic color restoration system according to claim 6, characterized in that: The repair processing module specifically includes: Object recognition and coloring information acquisition unit: receives the colored object information from the color mapping module and identifies the object's area, boundaries, and missing texture details; Image restoration region analysis unit: Determines the edge position of an object based on an edge detection algorithm, identifies the object's outline, checks for missing textures and details, and supplements them with filling and interpolation algorithms; Detail recovery and texture restoration unit: Restores these details based on the texture restoration algorithm, fills the image based on the surrounding pixels, and is used to restore texture and color details, enhance areas with missing or blurred textures, and increase the clarity of details; Color restoration and adjustment unit: performs local color adjustments based on the object's ambient lighting, shadow areas, and other contextual information, adjusting the global and local color balance of the image; Detail enhancement and lighting restoration unit: performs lighting restoration based on the lighting conditions on the object surface, restores natural lighting effects, and adjusts the brightness and contrast of the image through HDR technology.

8. The deep learning-based movie image object recognition and automatic color restoration system according to claim 7, characterized in that: The image restoration region analysis unit specifically includes: The edge position of the object is determined based on the edge detection algorithm, and the object contour is identified. The edge detection algorithm formula is: ； Where, Indicates the gradient size at a certain position in the image, Represents the horizontal gradient at a certain position in the image, Represents the vertical gradient of a certain position in the image, represents the gradient direction at a certain position in the image, is a mathematical function.

Images