Video image enhancement restoration method and system adopting deep learning

Abstract

The invention provides a video image enhancement restoration method and system adopting deep learning, and the method comprises the steps: firstly detecting user input in real time, and judging whether a user inputs an image processing keyword or not; secondly, when it is detected that a user inputs an image processing keyword, keyword feature vectors and video image sequence feature vectors are connected end to end and then input to a trained first convolutional neural network model for processing; thirdly, when the keyword input by the user is not detected, inputting the video image sequence feature vector into a trained second convolutional neural network model for processing; and finally, the first convolutional neural network model carries out processing or the second convolutional neural network model outputs the processed video image to a display module. According to the method, the first convolutional neural network model and the second convolutional neural network model are adopted, targeted processing is performed according to keyword conditions input by the user, the accuracy and efficiency of video processing are greatly improved, and the user experience is greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of video image restoration processing, and in particular to a video image enhancement and restoration method and system using deep learning. Background Art

[0002] With the rapid development of information technology, video content has become an important source of information and entertainment for people. However, in all aspects of video acquisition, processing, transmission and display, due to various factors such as poor lighting conditions, equipment limitations, compression algorithms, etc., video quality often deteriorates, manifested as image blur, noise, color distortion and other problems. These problems seriously affect the usability and viewing experience of video images. Traditional video image enhancement methods mainly include sharpening, contrast adjustment, denoising and other techniques, which are often based on simple linear processing models such as convolution filters and frequency domain techniques. Although these methods can improve image quality to a certain extent, they usually cannot handle the restoration of image details well, especially when the image is severely damaged. In addition, these traditional methods are computationally intensive when processing high-resolution videos and are difficult to process in real time, which limits their scope of application.

[0003] With the development of deep learning technology, especially the successful application of convolutional neural networks (CNN) in the field of image recognition and processing, new ideas and methods have been provided to solve the problem of video image enhancement. Deep learning models can learn complex features and patterns of images through a large amount of data, thereby effectively enhancing and repairing images. This method can dynamically adjust the processing strategy and intelligently optimize the processing effect according to different image content and damage conditions, greatly improving the processing quality and efficiency.

[0004] However, although deep learning has shown superior performance in the field of image processing, existing deep learning methods still face many challenges in video image enhancement applications. For example, most models require a large amount of training data and expensive computing resources, and lack support for real-time processing. In addition, these models often lack the ability to adapt to specific image processing tasks, such as being unable to dynamically adjust activation functions or other network parameters based on the specific characteristics of the video content. In addition, existing automatic processing rarely considers user input, and does not make adjustments based on the actual needs of users after automatic processing. During intelligent processing, artificial intelligence processing methods cannot perform targeted adaptive adjustments based on the brightness or other features of the image. In view of the limitations of existing video processing, refined, automated, real-time, and targeted video image processing methods urgently need a new solution to improve processing efficiency and accuracy, which is beneficial to improving user satisfaction. Summary of the invention

[0005] In response to the above problems mentioned in the prior art, the present invention provides a video image enhancement and restoration method and system using deep learning. The method first detects user input in real time to determine whether the user has input an image processing keyword; secondly, when it is detected that the user has input an image processing keyword, the keyword is encoded and converted into a keyword feature vector, and the keyword feature vector and the video image sequence feature vector are connected end to end and input into a trained first convolutional neural network model for processing; thirdly, when the user input keyword is not detected, the video image sequence feature vector is input into a trained second convolutional neural network model for processing; finally, the first convolutional neural network model processes or the second convolutional neural network model outputs the processed video image to a display module. The present application adopts the first convolutional neural network model and the second convolutional neural network model to perform targeted processing according to the user input keyword conditions, which greatly improves the accuracy and efficiency of video processing and greatly increases the user experience.

[0006] The present application provides a video image enhancement and restoration method using deep learning, comprising the steps of:

[0007] S1: Real-time detection of user input to determine whether the user has input an image processing keyword;

[0008] S2: When it is detected that the user inputs an image processing keyword, the keyword is encoded and converted into a keyword feature vector, and the keyword feature vector and the video image sequence feature vector are connected end to end and input into the trained first convolutional neural network model for processing;

[0009] S3: When no user input keyword is detected, the video image sequence feature vector is input into the trained second convolutional neural network model for processing;

[0010] S4: The first convolutional neural network model processes or the second convolutional neural network model outputs the processed video image to the display module.

[0011] The present application discloses a video image enhancement and restoration method using deep learning, wherein the trained first convolutional neural network model uses an improved Mish activation function f(x):

[0012] f(x)=x*tanh(ln(1+e βx ))

[0013] Among them, β is a parameter dynamically adjusted according to the keywords input by the user; tanh is the hyperbolic tangent function; ln represents the natural logarithm; and x is the output of the convolutional layer.

[0014] Preferably, the image processing keywords include: one of: noise reduction, sharpening, color correction, contrast enhancement, and detail enhancement.

[0015] Preferably, the trained first convolutional neural network model is trained by using a dataset of lossy and lossless images with keywords and labeled pairs; the trained second convolutional neural network model is trained by using a dataset of lossy and lossless images with labeled pairs.

[0016] Preferably, the trained second convolutional neural network model adopts an improved Sigmoid activation function:

[0017]

[0018] Among them, Z is the convolution operation output of the convolution layer of the second convolutional neural network model, R1 is the average brightness of the input video frame image; γ is the brightness adjustment parameter.

[0019] The present application also provides a video image enhancement and restoration system using deep learning, comprising:

[0020] The detection module detects user input in real time and determines whether the user has input image processing keywords;

[0021] The first neural network model processing module, when detecting that the user inputs an image processing keyword, encodes the keyword and converts it into a keyword feature vector, and connects the keyword feature vector and the video image sequence feature vector end to end and inputs them into the trained first convolutional neural network model for processing;

[0022] The second convolutional neural network model processing module inputs the video image sequence feature vector into the trained second convolutional neural network model for processing when no user input keyword is detected;

[0023] The output module, the first convolutional neural network model processes or the second convolutional neural network model outputs the processed video image to the display module.

[0024] Preferably, the trained first convolutional neural network model adopts an improved Mish activation function f(x):

[0025] f(x)=x*tanh(ln(1+e βx ))

[0026] Among them, β is a parameter dynamically adjusted according to the keywords input by the user; tanh is the hyperbolic tangent function; ln represents the natural logarithm; and x is the output of the convolutional layer.

[0027] Preferably, the image processing keywords include: one of: noise reduction, sharpening, color correction, contrast enhancement, and detail enhancement.

[0028] Preferably, the trained first convolutional neural network model is trained by using a dataset of lossy and lossless images with keywords and labeled pairs; the trained second convolutional neural network model is trained by using a dataset of lossy and lossless images with labeled pairs.

[0029] Preferably, the trained second convolutional neural network model adopts an improved Sigmoid activation function:

[0030]

[0031] Among them, Z is the convolution operation output of the convolution layer of the second convolutional neural network model, R1 is the average brightness of the input video frame image; γ is the brightness adjustment parameter.

[0032] The present invention provides a video image enhancement and restoration method and system using deep learning, which can achieve the following beneficial technical effects:

[0033] 1. The present invention provides a video image enhancement and restoration method and system using deep learning. The method first detects user input in real time to determine whether the user has input an image processing keyword; secondly, when the user inputs an image processing keyword, the keyword is encoded and converted into a keyword feature vector, and the keyword feature vector and the video image sequence feature vector are connected end to end and input into a trained first convolutional neural network model for processing; thirdly, when the user input keyword is not detected, the video image sequence feature vector is input into a trained second convolutional neural network model for processing; finally, the first convolutional neural network model processes or the second convolutional neural network model outputs the processed video image to a display module. The present application adopts the first convolutional neural network model and the second convolutional neural network model to perform targeted processing according to the user input keyword conditions, which greatly improves the accuracy and efficiency of video processing and greatly increases the user experience.

[0034] 2. The first convolutional neural network model trained by the present invention adopts an improved Mish activation function, and adds the value of β as a consideration into the calculation process of the activation function, which greatly enhances the pertinence of video processing to meet customer needs. By setting image processing keywords including: noise reduction, sharpening, color correction, contrast enhancement, and detail enhancement, and dynamically adjusting the value of the parameter according to the keyword input by the user, the value is determined according to the keyword, and the correspondence between each keyword and the value is set through multiple training experiences, which greatly improves the accuracy and comprehensiveness of video processing. The video image enhancement and restoration method of the present invention utilizes an improved deep learning model, especially by introducing a dynamically adjusted activation function, so that the processing process can be automatically adjusted according to the specific characteristics of the video content. This adaptive capability makes the method more accurate and effective when processing different types of image damage (such as blur, noise, color distortion, etc.), thereby significantly improving image quality and the user's visual experience.

[0035] 3. The second convolutional neural network model of this scheme adopts an improved Sigmoid activation function. In the activation function, the average brightness of the input video frame image is used as the reference condition for adjustment. When there is no user keyword prompt, the overall visual effect quality of the video image is further balanced. The convolutional neural network model used in the present invention is optimized and designed to quickly respond to and process high-resolution video streams. This high-efficiency processing capability is particularly suitable for real-time video applications, such as live video broadcasting, real-time monitoring and other scenarios, which meets the strict requirements of modern video applications for speed and efficiency, and greatly improves user satisfaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1 It is a schematic diagram of the steps of a video image enhancement and restoration method using deep learning of the present invention;

[0038] Figure 2 It is a schematic diagram of a video image enhancement and restoration system using deep learning according to the present invention. DETAILED DESCRIPTION

[0039] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] Embodiment 1:

[0041] In view of the above problems mentioned in the prior art, in order to solve the above technical problems, as shown in the attached Figure 1 As shown: This application provides a video image enhancement and restoration method using deep learning, comprising the steps of:

[0042] S1: Real-time detection of user input, judging whether the user has input of image processing keywords; in the video image enhancement and restoration method of the present invention, real-time detection of user input keywords is realized by a user interface (UI) integrated in the video processing system, which allows the user to enter keywords describing video processing requirements through a text input box, such as "noise reduction", "sharpening", "color correction", etc. After the user inputs, the system immediately parses and processes the input text to identify the keywords contained therein; natural language processing (NLP) technology, especially text classification and keyword extraction algorithms, is used to analyze the user input. This can be based on a predefined keyword list or use a machine learning model to identify and understand the user's intention. The system is configured with an event listener, which immediately triggers the analysis process once the user's input activity, such as a keystroke, is detected. This ensures that the system can respond to the user's input without delay and provide a real-time interactive experience. The user interface is designed to be simple and intuitive, providing clear input prompts and instant feedback to ensure that users can easily submit their processing requirements. The interface can also provide automatic completion and suggestion functions to help users quickly find and select the image processing keywords they need.

[0043] In one embodiment, an online video editing platform where users can upload their videos and select different image enhancement options to apply in real time, the user uploaded a video shot in a low-light environment, which is blurry and has distorted colors. The user enters "improve clarity, color correction" into the input box and submits it. The system parses the user's input in real time and recognizes that "improve clarity" corresponds to "sharpening" processing, while "color correction" directly corresponds to the defined keywords. The system then automatically associates these keywords with the corresponding image processing algorithms and applies them to the video uploaded by the user. The processed video is instantly displayed on the user interface, and the user can compare the original and processed videos to evaluate the effect of the image enhancement.

[0044] S2: When it is detected that the user inputs an image processing keyword, the keyword is encoded and converted into a keyword feature vector, and the keyword feature vector is connected end to end with the video image sequence feature vector and then input into the trained first convolutional neural network model for processing; in one embodiment, an advanced video editing software that allows users to select and apply image enhancement technology in real time to improve video quality. The user enters image processing requirements such as "remove noise" and "enhance details" through the text input box of the graphical user interface (GUI). The system monitors these inputs in real time and captures the text immediately. The keyword encoding system uses an embedding layer (usually a pre-trained word embedding model such as Word2Vec or GloVe) to convert the captured keywords into dense feature vectors. For example, "remove noise" may be converted to a point in a high-dimensional space. These word vectors capture the semantic attributes of the keywords, allowing the network to understand their meaning in image processing. At the same time, the system pre-processes the uploaded video, such as frame extraction, color space conversion, etc., to generate a feature vector of the video image sequence. It also includes using other deep learning models (such as pre-trained CNN) to extract features for each frame. The keyword feature vector is combined with the feature vector of the video image sequence for each frame. This is usually done by concatenating the vectors to form an extended feature vector that not only contains the visual features of the image, but also incorporates the user's processing intent. The combined feature vector is fed into a pre-trained convolutional neural network (CNN) that has been trained to perform corresponding image enhancement tasks, such as noise reduction and detail enhancement. The network uses the input feature vector for forward propagation and applies its learned filters and weights to enhance the video image. The processed video image sequence is updated and displayed in real time in the user interface, and the user can immediately see the enhanced video effect and make further adjustments or confirm the processing results as needed.

[0045] In one embodiment, feature vectors are extracted from each video frame for subsequent object recognition, tracking, and behavior analysis. First, continuous frames are captured from the video stream, and each frame is preprocessed, including resizing (such as resizing all frames to a uniform 256x256 pixels), normalization (scaling pixel values ​​to between 0 and 1), and color space conversion (such as converting from RGB to grayscale) to fit the trained model. A suitable pre-trained convolutional neural network model is selected for feature extraction, such as VGG16, ResNet, or MobileNet. These models perform well on multiple visual tasks and can extract rich image features. For example, the VGG16 model is used, which has a large depth and can capture complex visual patterns, suitable for complex scenes in traffic monitoring. The preprocessed image is input into the selected CNN model. The last few layers of the network (such as the fully connected layers) are usually not used, but the output of the intermediate layers is used as the feature vector. This is because the intermediate layers usually contain more general image features that are sufficient to describe the main objects and structures in the image. After each frame of the image passes through the network, the feature vector is extracted from the specified intermediate layer. For example, the feature map extracted from the last convolutional layer of VGG16. The extracted feature vector can be directly used in machine learning models such as support vector machines (SVM) or deep learning models for vehicle and pedestrian detection and classification. The feature vector can also be used to calculate the similarity between video frames to support object tracking analysis.

[0046] In one embodiment, the system allows users to specify the type of image processing they want to perform by entering keywords, such as "noise reduction", "sharpening", and "color correction". The system needs to convert these input keywords into feature vectors so that the subsequent convolutional neural network can use these vectors for more targeted image processing. According to system requirements and resources, select a suitable word embedding model. For example, Word2Vec is selected because it is simple, efficient, and suitable for capturing the semantic relationship between words. Assume that we use Google's pre-trained Word2Vec model, which is trained on large-scale text data. The keywords entered by the user may require some preprocessing, including conversion to lowercase, removal of stop words and punctuation, etc., to ensure that the input format is consistent with the training data of the word embedding model. The processed keywords are input into the Word2Vec model to obtain the corresponding word vector. For example, input "noise reduction" to obtain a 300-dimensional vector, which represents the semantic information of "noise reduction" in a high-dimensional space. The obtained keyword feature vector can then be combined with the feature vector of the video image sequence. The combination method can be a series connection of the vectors, that is, directly connecting the keyword vector and the image feature vector end to end to form a longer vector for subsequent convolutional neural network processing. The combined feature vectors are input into the neural network, which performs specific image processing tasks based on these features, such as noise suppression based on the "denoising" vector.

[0047] S3: When no user input keyword is detected, the feature vector of the video image sequence is input into the trained second convolutional neural network model for processing; in one embodiment, a video surveillance analysis platform needs to process the video streams from multiple cameras in real time to ensure that the image quality meets certain standards for subsequent monitoring and analysis. Users usually do not enter specific processing keywords in this scenario, and the system needs to automatically perform image enhancement tasks. The video stream is received in real time from the connected camera. Each video frame first undergoes basic preprocessing, including color correction and resizing, to adapt to the input requirements of the network model. Then, the system extracts features from the video frame. This step can be implemented using some standard image processing algorithms or shallower convolutional networks, with the goal of extracting useful visual features from the original video frame. After preprocessing and feature extraction, the feature vector of the video image sequence will be input into the pretrained second convolutional neural network model. This model is specifically designed for general image enhancement, such as automatic adjustment of contrast, sharpening, and slight noise removal, to adapt to general application scenarios without specific enhancement targets. The model can be a classic deep learning architecture, such as U-Net or VGG, which excel in image reconstruction and enhancement tasks. The model is trained with historical data and can handle a variety of common image quality issues. The processed video stream will be re-encoded and sent back to the server at the monitoring center or directly output to the monitoring display. Users can observe significantly improved image quality even if they do not specify specific enhancement requirements. The system can automatically optimize image quality without user interaction, which is critical for application scenarios that require high efficiency and automation, such as public safety monitoring or traffic flow monitoring systems. In addition, the automated processing flow reduces dependence on expertise and improves the usability and practicality of the system.

[0048] S4: The first convolutional neural network model processes or the second convolutional neural network model outputs the processed video image to the display module. In one embodiment, the video image enhancement and restoration method is applied to the urban traffic monitoring system, the purpose of which is to improve the clarity and visibility of the real-time video captured by the traffic camera, so as to better monitor and manage urban traffic. The video stream captured by the video processing traffic camera in real time first enters the system, and the system automatically selects appropriate processing keywords (such as "increase brightness", "defogging") according to the current environmental conditions (such as night, rainy days). The keywords are encoded as feature vectors and combined with the feature vectors of the video image sequence. These combined feature vectors are then input into the first convolutional neural network model for processing. The video after image enhancement processing is output in real time to the large display screen of the traffic management center. This allows traffic management personnel to effectively direct traffic and respond quickly to various situations based on clear video monitoring of real-time traffic conditions.

[0049] In one implementation example, an online video content providing platform, considering that the present invention can also be applied to an online video platform, is used to improve the quality of videos uploaded by users and enhance the viewing experience. After being uploaded to the platform, the original video content uploaded by the user is automatically processed by a second convolutional neural network model, which is optimized for general image quality problems (such as slight blur, color correction). In this case, since the user does not specify specific processing requirements, the system will apply a set of standard image enhancement strategies to ensure that the video quality meets the platform standards. The processed video is visible in the user's account and can be played directly. The platform can also provide a comparison view to allow users to see the effect comparison before and after the enhancement, and ensure that it is satisfied before publishing. When viewers access these videos, they can enjoy a clearer and higher-quality visual experience, which enhances the attractiveness and user satisfaction of the platform.

[0050] In some real-time examples, the trained first convolutional neural network model adopts an improved Mish activation function f(x):

[0051] f(x)=x*tanh(ln(1+e βx ))

[0052] Among them, β is a parameter dynamically adjusted according to the keywords input by the user; tanh is the hyperbolic tangent function; ln represents the natural logarithm; and x is the output of the convolution layer. In one embodiment, the first convolutional neural network model usually adopts a deep architecture to extract complex image features. The following are its main components: Input layer: Receives a preprocessed video image sequence, which may have undergone basic processing such as resizing and normalization. Convolutional layer and activation layer: Multiple convolutional layers are arranged alternately, and each convolutional layer is followed by an activation layer. The convolutional layer is used to extract the features of the image, and the activation layer after each convolutional layer uses an improved Mish activation function to enhance the nonlinear processing ability of the model and retain more image details. Each convolution layer may use filters of different sizes and step sizes to meet the requirements of feature extraction at different levels. Pooling layer: Pooling layers (such as maximum pooling or average pooling) are used after some convolutional layers to reduce the spatial dimension of the data, improve computational efficiency, and retain important features. Normalization layer: Use batch normalization layers to stabilize the learning process, accelerate convergence, and reduce internal covariate shift by normalizing the layer output. Fully connected layers and output layers: The last few layers of the network are fully connected layers. The final output layer may be designed as a regression layer (directly output pixel values) or a classification layer (output image processing type) depending on the specific task.

[0053] In one embodiment, an online photo editing tool is used that allows users to upload images and select the type of enhancement they need, such as "enhance details" or "color correction." The user uploads a travel photo with distorted colors and unclear details. The image is first preprocessed, resized, and normalized. The preprocessed image enters the first convolutional neural network, and after a series of convolution, activation, and pooling operations, the image features are gradually extracted and enhanced. After being processed by multiple levels of the network, the color of the image is corrected and the details are clearer. The processed image is output through the output layer of the network and displayed to the user, who can compare the original image with the processed effect to confirm satisfaction.

[0054] In some real-time examples, the image processing keywords include: one of noise reduction, sharpening, color correction, contrast enhancement, and detail enhancement.

[0055] In some real-time examples, the value of parameter β is dynamically adjusted according to the keyword input by the user, wherein the β value is determined according to the keyword, and the corresponding relationship between each keyword and the β value is as follows:

[0056] Keywords Beta value Noise Reduction 0.5 Sharpen 2 Color Correction 1.2 Contrast Enhancement 1.8 Detail Enhancement 2.5 .

[0057] In some implementation examples, determining the correspondence between keywords and β values ​​is a multifaceted process based on experimental data, theoretical research, and professional experience. The following are general steps and methods for determining these β values: First, a deep understanding of the image processing task represented by each keyword is required. For example, denoising focuses on reducing random noise in an image while preserving details, while sharpening increases the local contrast of an image to highlight edges and details. Each process has different effects on image characteristics, so the requirements for activation functions will also be different. The role of the activation function, considering the role of β values ​​in the activation function, is usually to control the shape of the function's curve, thereby affecting the degree and speed of activation. Larger β values ​​make the activation function curve steeper, which is particularly useful in tasks such as sharpening or detail enhancement that require strengthening specific frequency content. In contrast, smaller β values ​​make the curve smoother, suitable for scenarios such as denoising that require more gentle processing. Experimental tuning Determining β values ​​usually involves a lot of experimental and tuning processes: Data preparation: Prepare a dataset containing a variety of image quality issues, each corresponding to one or more keywords. Model training and testing: Train the model using different β values ​​and evaluate its performance on the validation set. By observing the impact of different β values ​​on the results, the optimal value can be found. Iterative optimization: Adjust the β value based on the test results and test repeatedly until the best setting for each task is found.

[0058] In some real-time examples, the trained first convolutional neural network model is trained by using a dataset of lossy and lossless images with keywords and labeled pairs; the trained second convolutional neural network model is trained by using a dataset of lossy and lossless images with labeled pairs.

[0059] In some real-time examples, the trained second convolutional neural network model uses an improved Sigmoid activation function:

[0060]

[0061] Among them, Z is the output of the convolution operation of the convolution layer of the second convolutional neural network model, R1 is the average brightness of the input video frame image; γ is the brightness adjustment parameter. The traditional Sigmoid activation function is a smooth nonlinear activation function used in neural networks, which compresses the input value between 0 and 1. This function was widely used in early neural networks for binary classification problems. However, when processing images with complex lighting conditions, the traditional Sigmoid function may not be flexible enough because its response to the input value is fixed. In the improved Sigmoid activation function mentioned above, by adding the average brightness R1 of the video frame image and a brightness adjustment parameter γ to the function, the response of the activation function can be dynamically adjusted according to the brightness conditions of the image. Specifically, this design allows the network to automatically adjust its activation threshold when processing too bright or too dark images to adapt to different lighting environments. By combining the average brightness of the image, the threshold of the activation function can be automatically adjusted up or down according to the overall brightness level of the image. This means that for darker images, the activation threshold will be lowered, making the network easier to activate; for brighter images, the threshold will be increased, thereby suppressing over-activation. This adaptive mechanism makes the network more robust when processing images under different lighting conditions, and can maintain the natural appearance of image details and contrast. For each frame of the input video image, its average brightness R1 is calculated, which can be obtained by averaging all pixel values. The brightness adjustment parameter r is pre-set or automatically learned according to the needs of experiments or practical applications, which determines how the average brightness affects the adjustment of the activation threshold.

[0062] In some embodiments, the second convolutional neural network model is designed for general image enhancement tasks and is suitable for scenarios without specific user input. The following are its main components: Input layer: Receives pre-processed video image data from the front end, which may have been processed by color space conversion, normalization, etc. to meet the network input requirements. Convolution layer and activation layer: Contains multiple convolution layers for extracting image features. Each convolution layer is followed by an improved Sigmoid activation layer, which improves the flexibility and adaptability of activation by introducing dynamic adjustment parameters to adapt to different image contents. Convolution layers are usually equipped with filters of different sizes to capture image features from coarse to fine. Pooling layer: Use maximum pooling or average pooling layers to reduce the spatial size of feature maps, help reduce the amount of calculation, while retaining key information. Normalization layer: Batch normalization may be used after some convolution layers, which helps stability and efficiency during network training. Fully connected layer: The back end of the network contains several fully connected layers to integrate the learned features and output the final image enhancement results. The output layer may regress to specific image pixels or output certain parameters of image processing, depending on the specific application requirements.

[0063] In one embodiment, the second convolutional neural network model is applied to a smart home security system, which is used to automatically enhance the video images received from the home security camera so that the user can monitor the situation at home more clearly. The system receives the video stream from the camera at home in real time. The video first undergoes basic preprocessing, including adjusting the resolution and normalization. The preprocessed video data enters the second convolutional neural network. The network processes the video through a series of convolution and pooling layers, using a modified Sigmoid activation function to activate after each convolution layer. The activation function adjusts the activation parameters according to the characteristics of the real-time video, such as brightness or contrast levels. After being processed by the network, the video image is significantly improved in contrast and clarity. The enhanced video is displayed instantly on the user's mobile phone or other device, and the user can clearly see all the details, such as faces or other important security features.

[0064] This application also provides a video image enhancement and restoration system using deep learning, such as Figure 2As shown, it includes: Video input device Camera: As a video source, it can be a closed-circuit television (CCTV) camera, a webcam, or any other type of video capture device for capturing video data in real time. Video receiving card / device: Converts the analog signal captured by the camera into a digital signal (if the camera outputs an analog signal), or directly receives a digital video stream. Processing unit: The computing server is equipped with high-performance CPUs and GPUs for processing complex deep learning models and algorithms. The server may contain multiple GPUs to support high-concurrency video data processing. Dedicated deep learning accelerators (such as NVIDIA Tesla GPUs or Google TPUs): Provide additional processing power, especially hardware optimized for deep learning tasks. High-speed storage systems: Such as solid-state drives (SSDs) or network-attached storage (NAS), used to store video data, model weights, and other necessary data. Routers and switches are used for the connection and data transmission of devices in the internal network to ensure the fast transmission of video data and processing results. The monitoring display is used to display the processed video images in real time, which can be a single large screen or a multi-screen display system. Mobile devices and terminals: Such as tablets and smartphones, used to remotely access and control the system and view video output. The camera is connected to the network router via Ethernet or Wi-Fi, and the video data is transmitted to the computing server over the network. In the case of an analog camera, the video receiving card converts the analog signal into a digital format and transmits it in the same way. The GPU and other processing units are installed directly on the server motherboard or connected through a PCIe expansion slot. Storage devices are connected to the server through high-speed interfaces such as SATA or NVMe to ensure data transmission speed. The server is connected to the monitoring display via HDMI or DisplayPort to display the processed video. For remote access, the server transmits the video stream to a mobile device or remote computer over the network. All devices (including servers, cameras, and display devices) are connected through routers and switches to ensure the stability and security of data communication.

[0065] The detection module detects user input in real time and determines whether the user has input image processing keywords;

[0066] The first neural network model processing module, when detecting that the user inputs an image processing keyword, encodes the keyword and converts it into a keyword feature vector, and connects the keyword feature vector and the video image sequence feature vector end to end and inputs it into the trained first convolutional neural network model for processing; Generation of keyword feature vector: In one embodiment, the keyword input by the user is converted into a fixed-length feature vector through an embedding model (such as Word2Vec or GloVe). For example, the keyword "denoising" may be converted into a 256-dimensional vector. Extracting image sequence feature vectors from videos usually uses a pre-trained convolutional neural network (such as VGG16 or ResNet). After each frame of the image is processed by the network, feature vectors are extracted from a certain intermediate layer, and these vectors may also be 256-dimensional. The keyword feature vector is concatenated with the feature vector of each frame of the image. This is usually achieved by directly operating the vector array in programming. For example, using the NumPy library in Python, if the keyword vector is kv and the image feature vector is iv, the concatenation operation is: cv = ((kv, iv)). The concatenated vector will have a higher dimension. For example, if both vectors are 256-dimensional, the concatenated vector will be 512-dimensional. The concatenated vector is input into another neural network (such as the first convolutional neural network model) for actual video processing tasks, such as image denoising, sharpening, etc. The network processing results can be used to improve video quality, output to a display module, or stored in a database.

[0067] The second convolutional neural network model processing module inputs the video image sequence feature vector into the trained second convolutional neural network model for processing when no user input keyword is detected;

[0068] The output module, the first convolutional neural network model processes or the second convolutional neural network model outputs the processed video image to the display module.

[0069] In some real-time examples, the trained first convolutional neural network model adopts an improved Mish activation function f(x):

[0070] f(x)=x*tanh(ln(1+e βx ))

[0071] Among them, β is a parameter that is dynamically adjusted according to the keywords entered by the user; tanh is the hyperbolic tangent function; ln represents the natural logarithm; and x is the output of the convolutional layer. The original Mish activation function is a smooth non-monotonic activation function that has been proven to be superior to the traditional ReLU function in a variety of deep learning tasks because it can maintain good gradients and reduce the problem of gradient vanishing. The Mish function is often used to promote the effective flow of information in deep networks. In this application, the Mish function enhances its flexibility by adding an adjustable parameter b. The value of this parameter β is adjusted according to the keywords entered by the user so that the response of the activation function better meets specific image processing tasks, such as sharpening, noise reduction, etc. Parameter adjustment mechanism: According to the keywords specified by the user, such as "noise reduction" or "sharpening", the parameter β will be set to different values. This is because different image processing tasks may require the activation function to respond to the input data in different ways: For "sharpening": a larger β value is required to make the activation function more sensitive, which can enhance the details and edges in the image and make the image look clearer. For "noise reduction": a smaller β value is required to make the curve of the activation function smoother, which helps to suppress the noise of the image and improve the overall visual quality of the image. By dynamically adjusting the β value, the activation function can adapt to different image processing requirements more flexibly, thereby obtaining better processing results on specific tasks. For example, when it is necessary to highlight the details of the image, by increasing the β value, the network can emphasize the high-frequency part of the image more, thereby better recovering the details. Conversely, reducing the β value can help the network reduce high-frequency noise while retaining the main structure of the image. Users can directly affect the results of image processing through simple keyword input, which makes the operation of the system more intuitive and easy to master. Dynamic adjustment of β can be achieved in several ways: pre-define a β value for each keyword, which can be obtained through experimental testing and optimization. Use an auxiliary neural network to dynamically predict the best value of β based on the input keyword and image content. Provide a user interface that allows users to manually adjust the β value as needed, and the system optimizes and learns based on this feedback.

[0072] In some real-time examples, the image processing keywords include: one of noise reduction, sharpening, color correction, contrast enhancement, and detail enhancement. In one embodiment, a vocabulary is used to define synonyms or near-synonyms for each preset keyword. For example, for "sharpening", synonyms can be "enhancing edges", "improving clarity", etc. This vocabulary can be created manually or automatically expanded using natural language processing tools such as WordNet or other semantic networks. NLP technology is used to understand the natural language input by the user. A text classification algorithm can be used to classify the user input to the closest preset keyword. A machine learning model (such as a support vector machine, a random forest, or a neural network) is used for training so that it can recognize and classify the user's input based on examples in the training data. If the system cannot determine the intention of the user's input, it can be designed to request the user to provide more information or select the closest processing type from a recommendation list. This method not only improves the accuracy of the system, but also increases the user's sense of participation, enabling the user to directly control the results of image processing. Synonyms and expressions related to each preset keyword are collected. A team of experts can be organized to define these keywords, or online resources and tools can be used to automatically generate relevant vocabulary. Collect or generate a dataset containing various possible user inputs, and annotate the correspondence between these inputs and preset keywords. Use this dataset to train a text classification model so that it can map user input to the closest preset keyword. In the user interface, when the user enters a keyword, the system can immediately feedback a confirmation message, asking the user whether it refers to a processing option that the system can provide. If the keyword entered by the user is unclear, the system can provide a drop-down menu or list for the user to select the option that best meets their needs.

[0073] In some real-time examples, the value of parameter β is dynamically adjusted according to the keyword input by the user, wherein the β value is determined according to the keyword, and the corresponding relationship between each keyword and the β value is as follows:

[0074] Keywords Beta value Noise Reduction 0.5 Sharpen 2 Color Correction 1.2 Contrast Enhancement 1.8 Detail Enhancement 2.5 .

[0075] The present invention provides a video image enhancement and restoration method and system using deep learning, which can achieve the following beneficial technical effects:

[0076] 1. The present invention provides a video image enhancement and restoration method and system using deep learning. The method first detects user input in real time to determine whether the user has input an image processing keyword; secondly, when the user inputs an image processing keyword, the keyword is encoded and converted into a keyword feature vector, and the keyword feature vector and the video image sequence feature vector are connected end to end and input into a trained first convolutional neural network model for processing; thirdly, when the user input keyword is not detected, the video image sequence feature vector is input into a trained second convolutional neural network model for processing; finally, the first convolutional neural network model processes or the second convolutional neural network model outputs the processed video image to a display module. The present application adopts the first convolutional neural network model and the second convolutional neural network model to perform targeted processing according to the user input keyword conditions, which greatly improves the accuracy and efficiency of video processing and greatly increases the user experience.

[0077] 2. The first convolutional neural network model trained by the present invention adopts an improved Mish activation function, and adds the value of β as a consideration into the calculation process of the activation function, which greatly enhances the pertinence of video processing to meet customer needs. By setting image processing keywords including: noise reduction, sharpening, color correction, contrast enhancement, and detail enhancement, and dynamically adjusting the value of the parameter according to the keyword input by the user, the value is determined according to the keyword, and the correspondence between each keyword and the value is set through multiple training experiences, which greatly improves the accuracy and comprehensiveness of video processing. The video image enhancement and restoration method of the present invention utilizes an improved deep learning model, especially by introducing a dynamically adjusted activation function, so that the processing process can be automatically adjusted according to the specific characteristics of the video content. This adaptive capability makes the method more accurate and effective when processing different types of image damage (such as blur, noise, color distortion, etc.), thereby significantly improving image quality and the user's visual experience.

[0078] 3. The second convolutional neural network model of this scheme adopts an improved Sigmoid activation function. In the activation function, the average brightness of the input video frame image is used as the reference condition for adjustment. When there is no user keyword prompt, the overall visual effect quality of the video image is further balanced. The convolutional neural network model used in the present invention is optimized and designed to quickly respond to and process high-resolution video streams. This high-efficiency processing capability is particularly suitable for real-time video applications, such as live video broadcasting, real-time monitoring and other scenarios, which meets the strict requirements of modern video applications for speed and efficiency, and greatly improves user satisfaction.

[0079] The above is a detailed introduction to a video image enhancement and restoration method and system using deep learning. This article uses specific examples to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the core idea of ​​the present invention; at the same time, for general technical personnel in this field, according to the ideas and methods of the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A video image enhancement and restoration method using deep learning, characterized in that: Includes steps: S1: Real-time detection of user input to determine whether the user has input an image processing keyword; S2: When it is detected that the user inputs an image processing keyword, the keyword is encoded and converted into a keyword feature vector, and the keyword feature vector and the video image sequence feature vector are connected end to end and input into the trained first convolutional neural network model for processing; S3: When no user input keyword is detected, the video image sequence feature vector is input into the trained second convolutional neural network model for processing; S4: The first convolutional neural network model processes or the second convolutional neural network model outputs the processed video image to the display module.

2. The video image enhancement and restoration method using deep learning as claimed in claim 1, characterized in that: The trained first convolutional neural network model adopts the improved Mish activation function f(x): f(x)=x*tanh(ln(1+e βx )) Among them, β is a parameter dynamically adjusted according to the keywords input by the user; tanh is the hyperbolic tangent function; ln represents the natural logarithm; and x is the output of the convolutional layer.

3. The video image enhancement and restoration method using deep learning as claimed in claim 2, characterized in that: The image processing keywords include: one of: noise reduction, sharpening, color correction, contrast enhancement, and detail enhancement.

4. The video image enhancement and restoration method using deep learning as claimed in claim 1, characterized in that: The trained first convolutional neural network model is trained by using a dataset of lossy and lossless images with keywords and labeled pairs; the trained second convolutional neural network model is trained by using a dataset of lossy and lossless images with labeled pairs.

5. The video image enhancement and restoration method using deep learning as claimed in claim 1, characterized in that: The trained second convolutional neural network model adopts an improved Sigmoid activation function: Among them, Z is the convolution operation output of the convolution layer of the second convolutional neural network model, R1 is the average brightness of the input video frame image, and γ is the brightness adjustment parameter.

6. A video image enhancement and restoration system using deep learning, characterized in that: include: The detection module detects user input in real time and determines whether the user has input image processing keywords; The first neural network model processing module, when detecting that the user inputs an image processing keyword, encodes the keyword and converts it into a keyword feature vector, and connects the keyword feature vector and the video image sequence feature vector end to end and inputs them into the trained first convolutional neural network model for processing; The second convolutional neural network model processing module inputs the video image sequence feature vector into the trained second convolutional neural network model for processing when no user input keyword is detected; The output module, the first convolutional neural network model processes or the second convolutional neural network model outputs the processed video image to the display module.

7. The video image enhancement and restoration system using deep learning as claimed in claim 6, characterized in that: The trained first convolutional neural network model adopts the improved Mish activation function f(x): f(x)=x*tanh(ln(1+e βx )) Among them, β is a parameter dynamically adjusted according to the keywords input by the user; tanh is the hyperbolic tangent function; ln represents the natural logarithm; and x is the output of the convolutional layer.

8. The video image enhancement and restoration system using deep learning as claimed in claim 6, characterized in that: The image processing keywords include: one of: noise reduction, sharpening, color correction, contrast enhancement, and detail enhancement.

9. The video image enhancement and restoration system using deep learning as claimed in claim 6, characterized in that: The trained first convolutional neural network model is trained by using a dataset of lossy and lossless images with keywords and labeled pairs; the trained second convolutional neural network model is trained by using a dataset of lossy and lossless images with labeled pairs.

10. The video image enhancement and restoration system using deep learning as claimed in claim 6, characterized in that: The trained second convolutional neural network model adopts an improved Sigmoid activation function: Among them, Z is the convolution operation output of the convolution layer of the second convolutional neural network model, R1 is the average brightness of the input video frame image, and γ is the brightness adjustment parameter.

Images