Video frame processing method and device, computer equipment and storage medium

Abstract

The invention belongs to the technical field of artificial intelligence, and relates to a video frame processing method, which comprises the following steps: acquiring a video frame from a video source; preprocessing the video frame to obtain a corresponding specified video frame; performing super-resolution processing on the specified video frame based on the target image processing model to generate a corresponding high-resolution video frame; wherein the target image processing model is a model generated by training an optimized specified diffusion model based on a pre-constructed super-resolution data set; performing image quality optimization processing on the high-resolution video frame to obtain a corresponding target video frame; and displaying the target video frame on a preset monitoring screen. The invention further provides a video frame processing device, computer equipment and a storage medium. In addition, the target video frame may be stored in a blockchain. According to the invention, the resolution of the video frame is efficiently and accurately improved based on the use of the target image processing model, so that the video imaging quality is improved.

Description

Technical Field

[0001] The present application relates to the fields of artificial intelligence and financial technology, and in particular to video frame processing methods, devices, computer equipment and storage media. Background Art

[0002] In the daily operation of financial enterprises, the application of monitoring systems is crucial. It not only helps enterprises ensure the security of their business, but also provides key evidence when abnormal situations occur. However, in actual applications, financial enterprises often face the problem of poor imaging effects of monitoring cameras, resulting in a large number of low-resolution video frames in the monitoring video. These frames are difficult to provide clear feature information, which seriously affects the effectiveness of the monitoring system.

[0003] In the monitoring system of financial enterprises, if the camera imaging effect is poor, a large number of low-resolution video frames will appear in the monitoring video. These video frames are difficult to provide clear feature information, making it impossible for the monitoring system to provide effective evidence when abnormal situations occur. For example, when a security incident occurs, if the key information in the monitoring video is unclear, it is impossible to accurately identify the identity or behavioral characteristics of the suspect, which brings great difficulties to the detection of the case. Therefore, in order to solve the problem of poor camera imaging in the monitoring system of financial enterprises, it is urgent to provide a solution that can improve the imaging quality of monitoring videos. Summary of the invention

[0004] The purpose of the embodiments of the present application is to propose a video frame processing method, apparatus, computer equipment and storage medium to solve the technical problem that the camera imaging effect in the monitoring system of existing financial enterprises is poor, resulting in a large number of low-resolution video frames in the monitoring video.

[0005] In order to solve the above technical problems, the embodiment of the present application provides a video frame processing method, which adopts the following technical solution:

[0006] Get video frames from a preset video source;

[0007] Preprocessing the video frame to obtain a corresponding designated video frame;

[0008] Performing super-resolution processing on the designated video frame based on a preset target image processing model to generate a corresponding high-resolution video frame; wherein the target image processing model is a model generated by training an optimized designated diffusion model based on a pre-constructed super-resolution data set;

[0009] Performing image quality optimization processing on the high-resolution video frame to obtain a corresponding target video frame;

[0010] The target video frame is displayed on a preset monitoring screen.

[0011] Furthermore, in performing super-resolution processing on the designated video frame based on the preset target image processing model to generate a corresponding high-resolution video frame, the method further includes:

[0012] Call the initial pre-trained diffusion model;

[0013] Obtaining a specified network structure of the pre-trained diffusion model;

[0014] Freezing the encoder layer of the specified network structure based on a preset freezing strategy, and fine-tuning the decoder part of the specified network structure based on a preset fine-tuning strategy to obtain a corresponding optimization model;

[0015] The optimized model is used as the specified diffusion model.

[0016] Furthermore, in performing super-resolution processing on the designated video frame based on the preset target image processing model to generate a corresponding high-resolution video frame, the method further includes:

[0017] Acquire a pre-constructed super-resolution dataset; wherein the super-resolution dataset includes image pairs with corresponding relationships between low resolution and high resolution;

[0018] Dividing the super-resolution dataset into a training set and a validation set;

[0019] determining an overall loss function corresponding to the specified diffusion model;

[0020] Based on the overall loss function and a preset optimization algorithm, the designated diffusion model is trained using the training set to obtain a trained first diffusion model;

[0021] Evaluate and optimize the first diffusion model based on the validation set to obtain a second diffusion model that meets preset performance requirements;

[0022] The second diffusion model is used as the target image processing model.

[0023] Furthermore, the step of evaluating and optimizing the first diffusion model based on the validation set to obtain a second diffusion model that meets the preset performance requirements specifically includes:

[0024] Performing evaluation processing on the first diffusion model based on the validation set to obtain a corresponding evaluation index score;

[0025] Analyze and process the evaluation index scores to obtain corresponding analysis results;

[0026] Adjusting the training strategy of the specified diffusion model based on the analysis result to obtain an adjusted target training strategy;

[0027] Retraining the first diffusion model based on the target training strategy, and repeating the evaluation and optimization process until a specified model that meets the performance requirements is obtained;

[0028] The designated model is used as the second diffusion model.

[0029] Furthermore, the step of preprocessing the video frame to obtain the corresponding designated video frame specifically includes:

[0030] Resizing the video frame to obtain a corresponding first video frame;

[0031] Performing cropping processing on the first video frame to obtain a corresponding second video frame;

[0032] Normalizing the second video frame to obtain a corresponding third video frame;

[0033] Performing format conversion processing on the third video frame to obtain a corresponding fourth video frame;

[0034] The fourth video frame is used as the designated video frame.

[0035] Furthermore, the step of performing image quality optimization processing on the high-resolution video frame to obtain a corresponding target video frame specifically includes:

[0036] Performing denoising processing on the high-resolution video frame to obtain a corresponding first designated video frame;

[0037] Performing contrast enhancement processing on the first designated video frame to obtain a corresponding second designated video frame;

[0038] Performing format conversion processing on the second designated video frame to obtain a corresponding third designated video frame;

[0039] Performing size restoration processing on the third designated video frame to obtain a corresponding fourth designated video frame;

[0040] The fourth designated video frame is used as the target video frame.

[0041] Furthermore, after the step of performing image quality optimization processing on the high-resolution video frame to obtain a corresponding target video frame, the method further includes:

[0042] Call the preset video file library;

[0043] Acquire a target video file corresponding to the video source from the video file library;

[0044] The target video frame is stored in the target video file.

[0045] In order to solve the above technical problems, the embodiment of the present application further provides a video frame processing device, which adopts the following technical solution:

[0046] A first acquisition module, used for acquiring video frames from a preset video source;

[0047] A preprocessing module, used for preprocessing the video frame to obtain a corresponding designated video frame;

[0048] A first processing module is used to perform super-resolution processing on the specified video frame based on a preset target image processing model to generate a corresponding high-resolution video frame; wherein the target image processing model is a model generated by training an optimized specified diffusion model based on a pre-constructed super-resolution data set;

[0049] A second processing module is used to perform image quality optimization processing on the high-resolution video frame to obtain a corresponding target video frame;

[0050] The display module is used to display the target video frame on a preset monitoring screen.

[0051] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0052] Get video frames from a preset video source;

[0053] Preprocessing the video frame to obtain a corresponding designated video frame;

[0054] Performing super-resolution processing on the designated video frame based on a preset target image processing model to generate a corresponding high-resolution video frame; wherein the target image processing model is a model generated by training an optimized designated diffusion model based on a pre-constructed super-resolution data set;

[0055] Performing image quality optimization processing on the high-resolution video frame to obtain a corresponding target video frame;

[0056] The target video frame is displayed on a preset monitoring screen.

[0057] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0058] Get video frames from a preset video source;

[0059] Preprocessing the video frame to obtain a corresponding designated video frame;

[0060] Performing super-resolution processing on the designated video frame based on a preset target image processing model to generate a corresponding high-resolution video frame; wherein the target image processing model is a model generated by training an optimized designated diffusion model based on a pre-constructed super-resolution data set;

[0061] Performing image quality optimization processing on the high-resolution video frame to obtain a corresponding target video frame;

[0062] The target video frame is displayed on a preset monitoring screen.

[0063] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0064] The present application first obtains a video frame from a preset video source; then preprocesses the video frame to obtain a corresponding designated video frame; then performs super-resolution processing on the designated video frame based on a preset target image processing model to generate a corresponding high-resolution video frame; subsequently performs image quality optimization processing on the high-resolution video frame to obtain a corresponding target video frame; and finally displays the target video frame on a preset monitoring screen. The present application obtains a designated video frame by preprocessing the video frame obtained from the video source, and then performs super-resolution processing on the designated video frame based on a target image processing model generated by training an optimized designated diffusion model based on a super-resolution data set to generate a corresponding high-resolution video frame, and further performs image quality optimization processing on the high-resolution video frame to obtain a corresponding target video frame, thereby achieving efficient and accurate improvement of the resolution of the video frame, which is beneficial to improving the quality of video imaging. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0066] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0067] Figure 2 A flowchart of an embodiment of a video frame processing method according to the present application;

[0068] Figure 3 is a structural schematic diagram of an embodiment of a video frame processing device according to the present application;

[0069] Figure 4 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0070] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0071] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0072] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0073] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0074] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0075] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to the laptop computer 1011, tablet computer 1012 or mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc.

[0076] The server 103 may be a server that provides various services, such as a background server that provides support for a web page displayed on the terminal device 101 .

[0077] It should be noted that the video frame processing method provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the video frame processing device is generally arranged in the server / terminal device.

[0078] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0079] Continue to refer Figure 2 , shows a flow chart of an embodiment of the video frame processing method according to the present application. According to different requirements, the order of the steps in the flow chart can be changed, and some steps can be omitted. The video frame processing method provided in the embodiment of the present application can be applied to any scenario that requires video processing, and the video frame processing method can be applied to products in these scenarios, for example, video processing in the financial insurance field. The video frame processing method comprises the following steps:

[0080] Step S201, obtaining a video frame from a preset video source.

[0081] In this embodiment, the video frame processing method is executed on an electronic device (eg Figure 1The server / terminal device shown in the figure) can obtain video frames through a wired connection or a wireless connection. It should be noted that the above-mentioned wireless connection method may include but is not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future. The executor of the present application may specifically be a monitoring system, or simply referred to as a system. The above-mentioned video source may be a surveillance camera, video file or other video input device used in a financial enterprise. Among them, video data can be captured frame by frame from the video source, and the video data is usually a video frame in the form of an image frame.

[0082] Step S202: pre-process the video frame to obtain a corresponding designated video frame.

[0083] In this embodiment, the above-mentioned specific implementation process of preprocessing the video frame to obtain the corresponding specified video frame will be further described in detail in subsequent specific embodiments of the present application, and will not be elaborated in detail here.

[0084] Step S203, performing super-resolution processing on the designated video frame based on the target image processing model to generate a corresponding high-resolution video frame; wherein the target image processing model is a model generated by training an optimized designated diffusion model based on a pre-constructed super-resolution dataset.

[0085] In this embodiment, the target image processing model can be loaded by using a deep learning framework. Then, by passing the specified video frame as input to the target image processing model, the input specified video frame is subjected to super-resolution processing by using the target image processing model to generate a high-resolution video frame, i.e., the above-mentioned high-resolution video frame. Among them, the construction process of the above-mentioned target image processing model will be further described in detail in the subsequent specific embodiments of this application, and will not be elaborated on here.

[0086] Step S205 , performing image quality optimization processing on the high-resolution video frame to obtain a corresponding target video frame.

[0087] In this embodiment, the above-mentioned specific implementation process of performing image quality optimization processing on the high-resolution video frame to obtain the corresponding target video frame will be further described in detail in subsequent specific embodiments of the present application, and will not be elaborated in detail here.

[0088] Step S206: display the target video frame on a preset monitoring screen.

[0089] In this embodiment, the optimized target video frame can be displayed in real time on a monitoring screen, wherein the monitoring screen is a pre-built screen for displaying monitored video frames.

[0090] The present application first obtains a video frame from a preset video source; then preprocesses the video frame to obtain a corresponding designated video frame; then performs super-resolution processing on the designated video frame based on a preset target image processing model to generate a corresponding high-resolution video frame; subsequently performs image quality optimization processing on the high-resolution video frame to obtain a corresponding target video frame; and finally displays the target video frame on a preset monitoring screen. The present application obtains a designated video frame by preprocessing the video frame obtained from the video source, and then performs super-resolution processing on the designated video frame based on a target image processing model generated by training an optimized designated diffusion model based on a super-resolution data set to generate a corresponding high-resolution video frame, and further performs image quality optimization processing on the high-resolution video frame to obtain a corresponding target video frame, thereby achieving efficient and accurate improvement of the resolution of the video frame, which is beneficial to improving the quality of video imaging.

[0091] In some optional implementations, before step S203, the electronic device may further perform the following steps:

[0092] Call the initial pre-trained diffusion model.

[0093] In this embodiment, the pre-trained diffusion model is a diffusion model that has been trained with a large-scale data set in an open source community, such as Stable Diffusion. The pre-trained diffusion model has been trained for image generation tasks on a large-scale data set and has good image generation capabilities. Among them, the loading tool or API provided by the pre-trained diffusion model can be used to ensure that the model structure and weights of the pre-trained diffusion model are correctly loaded into the memory.

[0094] Obtain a specified network structure of the pre-trained diffusion model.

[0095] In this embodiment, the above-mentioned specified network structure specifically refers to the U-Net network structure. The location of the U-Net network structure is found in the loaded pre-trained diffusion model. The U-Net network is the core denoising network of the diffusion model, which reconstructs the image in the latent space.

[0096] The encoder layer of the specified network structure is frozen based on a preset freezing strategy, and the decoder part of the specified network structure is fine-tuned based on a preset fine-tuning strategy to obtain a corresponding optimization model.

[0097] In this embodiment, most of the encoder layers (downsampling layers) of the U-Net network structure can be set to a non-trainable state, that is, the weights of these layers are frozen. Specifically, this can be achieved by setting the trainable attribute of these layers to False. Among them, the downsampling layer has learned good feature extraction capabilities during the pre-training process. In order to enable the model to handle super-resolution tasks, most of the encoder layers of the U-Net network structure are frozen. In addition, by determining the last 1-2 layers of the decoder part (up-sampling layer) of the U-Net network structure that needs fine-tuning. Then fine-tune the decoder part to obtain the corresponding optimization model.

[0098] The optimized model is used as the specified diffusion model.

[0099] In this embodiment, after obtaining the optimized model, the optimized model can be directly used as the above-mentioned designated diffusion model. In addition, according to the task requirements, the loss function in the fine-tuning process can be defined, such as L_UNet, to optimize the error between the potential representations. Specifically, it can include calculating the distance between the generated image and the real image in the latent space.

[0100] The present application calls the initial pre-trained diffusion model; then obtains the specified network structure of the pre-trained diffusion model; then freezes the encoder layer of the specified network structure based on a preset freezing strategy, and fine-tunes the decoder part of the specified network structure based on a preset fine-tuning strategy to obtain the corresponding optimization model; and subsequently uses the optimization model as the specified diffusion model. The present application obtains the specified network structure of the pre-trained diffusion model, and then freezes the encoder layer of the specified network structure based on the use of freezing strategy and fine-tuning strategy, and fine-tunes the decoder part of the specified network structure, so as to realize the intelligent and accurate construction of the optimized specified diffusion model. The present application retains the knowledge of the model pre-training stage by adjusting only a small number of parameters of the pre-trained diffusion model, which helps to avoid overfitting and maintain the generalization ability of the model.

[0101] In some optional implementations of this embodiment, before step S203, the electronic device may further perform the following steps:

[0102] A pre-constructed super-resolution dataset is obtained; wherein the super-resolution dataset includes image pairs with corresponding relationships between low resolution and high resolution.

[0103] In this embodiment, in order to optimize the model for the super-resolution task, a special super-resolution dataset is used. Specifically, image pairs containing low-resolution and high-resolution correspondences can be screened out from public datasets or self-collected images and used as the super-resolution dataset.

[0104] The super-resolution dataset is divided into a training set and a validation set.

[0105] In this embodiment, the super-resolution data set can be divided into a training set and a validation set according to a preset division ratio. There is no limitation on the selection of the division ratio, for example, 7:3 can be used.

[0106] An overall loss function corresponding to the specified diffusion model is determined.

[0107] In this embodiment, the specified diffusion model may also include a VAE (variational autoencoder) part. The above-mentioned overall loss function refers to the loss of VAE and U-Net, and the overall loss function is used to optimize the error between the generated image and the real image. Specifically, the overall loss function can be obtained by weighted summing the loss of VAE (KL divergence) and the loss of U-Net (L_UNet).

[0108] Based on the overall loss function and a preset optimization algorithm, the designated diffusion model is trained using the training set to obtain a trained first diffusion model.

[0109] In this embodiment, the above-mentioned specific algorithms such as Adam and SGD can be used. Specifically, model training is an iterative process, and the goal of model training is to minimize the overall loss function by adjusting the parameters of the model. During the training process, the model receives input data and generates prediction results. Then, the overall loss function is calculated to measure the difference between the predicted results and the actual results. After that, an optimization algorithm (such as gradient descent, Adam, etc.) is used to update the parameters of the model according to the gradient of the loss function. This process will be repeated until a certain stopping condition is reached (such as reaching a preset number of iterations, the loss function converges, etc.), and then the trained first diffusion model is obtained.

[0110] Among them, the incremental fine-tuning training method used requires adjusting all parameters of the model, which is computationally intensive and consumes more resources than full fine-tuning. Incremental fine-tuning only adjusts certain layers or specific parameters in the model, reducing the amount of computation and memory usage during training. And because only a part of the parameters are adjusted, incremental fine-tuning is usually faster than full fine-tuning, which is suitable for scenarios with limited resources or the need to quickly update the model.

[0111] In addition, the incremental fine-tuning training method can also reduce the risk of overfitting and maintain the generalization ability of the model trained with large-scale data. Full fine-tuning may cause the model to overfit to the new training data, especially when the amount of data is small. Incremental fine-tuning only adjusts a small number of parameters and retains the knowledge of the model pre-training stage, which helps to avoid overfitting and maintain the generalization ability of the model.

[0112] The first diffusion model is evaluated and optimized based on the verification set to obtain a second diffusion model that meets preset performance requirements.

[0113] In this embodiment, the above-mentioned specific implementation process of evaluating and optimizing the first diffusion model based on the verification set to obtain the second diffusion model that meets the preset performance requirements will be further described in detail in subsequent specific embodiments of the present application and will not be elaborated on here.

[0114] The second diffusion model is used as the target image processing model.

[0115] The present application obtains a pre-constructed super-resolution data set; then divides the super-resolution data set into a training set and a validation set; then determines the overall loss function corresponding to the specified diffusion model; subsequently, based on the overall loss function and a preset optimization algorithm, uses the training set to train the specified diffusion model to obtain a trained first diffusion model; further evaluates and optimizes the first diffusion model based on the validation set to obtain a second diffusion model that meets the preset performance requirements; and finally uses the second diffusion model as the target image processing model. The present application divides the acquired pre-constructed super-resolution data set into a training set and a validation set, and then, based on the determined overall loss function and a preset optimization algorithm, uses the training set to train the specified diffusion model to obtain a trained first diffusion model, and evaluates and optimizes the first diffusion model based on the validation set, so that the target image processing model that meets the preset performance requirements can be obtained intelligently and accurately, thereby improving the construction efficiency of the target image processing model. In addition, incremental fine-tuning of the specified diffusion model is achieved, which effectively optimizes the performance of the obtained target image processing model in the super-resolution task and ensures the model effect of the obtained target image processing model.

[0116] In some optional implementations, the evaluating and optimizing the first diffusion model based on the validation set to obtain a second diffusion model that meets preset performance requirements includes the following steps:

[0117] The first diffusion model is evaluated based on the validation set to obtain a corresponding evaluation index score.

[0118] In this embodiment, according to the task requirements, appropriate evaluation indicators are pre-selected, such as PSNR, SSIM, etc. Then, the super-resolution capability of the first diffusion model is evaluated using the validation set, and the scores of the generated images on indicators such as PSNR and SSIM, i.e., the above evaluation indicator scores, are calculated.

[0119] The evaluation index scores are analyzed and processed to obtain corresponding analysis results.

[0120] In this embodiment, based on the evaluation results, the deficiencies of the model are analyzed, such as detail loss, color distortion, etc., so as to obtain corresponding analysis results.

[0121] The training strategy of the designated diffusion model is adjusted based on the analysis result to obtain an adjusted target training strategy.

[0122] In this embodiment, the training strategies such as the number of frozen layers, the number of fine-tuned layers, and the learning rate of the specified diffusion model can be adjusted according to the generated analysis results to obtain an adjusted target training strategy.

[0123] The first diffusion model is retrained based on the target training strategy, and the evaluation and optimization process is repeated until a specified model that meets the performance requirements is obtained.

[0124] In this embodiment, the first diffusion model is retrained by using the adjusted target training strategy, and the evaluation and optimization process is repeated until the model achieves satisfactory performance, thereby obtaining a corresponding designated model.

[0125] The designated model is used as the second diffusion model.

[0126] The present application evaluates the first diffusion model based on the validation set to obtain the corresponding evaluation index score; then analyzes the evaluation index score to obtain the corresponding analysis result; then adjusts the training strategy of the specified diffusion model based on the analysis result to obtain the adjusted target training strategy; then retrains the first diffusion model based on the target training strategy, and repeats the evaluation and optimization process until the specified model that meets the performance requirements is obtained; finally, the specified model is used as the second diffusion model. The present application evaluates the first diffusion model based on the validation set to obtain the corresponding evaluation index score, and analyzes the evaluation index score to obtain the analysis result, and then adjusts the training strategy of the specified diffusion model based on the analysis result to obtain the adjusted target training strategy, and then retrains the first diffusion model using the target training strategy, and repeats the evaluation and optimization process until the second diffusion model with performance requirements can be obtained, thereby intelligently and accurately realizing the evaluation and optimization of the first diffusion model, and effectively optimizing the performance of the obtained second diffusion model in the super-resolution task.

[0127] In some optional implementations, step S202 includes the following steps:

[0128] The video frame is resized to obtain a corresponding first video frame.

[0129] In this embodiment, the resizing process refers to adjusting the captured video frame to the size required by the model input, which usually includes a scaling operation.

[0130] The first video frame is cropped to obtain a corresponding second video frame.

[0131] In this embodiment, the video frame may be cropped according to actual needs to remove unnecessary edge areas.

[0132] The second video frame is normalized to obtain a corresponding third video frame.

[0133] In this embodiment, the pixel values ​​of the video frame can be normalized to the range required by the model input (such as [0, 1] or [-1, 1]).

[0134] Perform format conversion processing on the third video frame to obtain a corresponding fourth video frame.

[0135] In this embodiment, the video frame can be converted from a common image format (such as BGR) to a format required by the model (such as RGB or grayscale).

[0136] The fourth video frame is used as the designated video frame.

[0137] In this embodiment, if the model supports batch processing, multiple video frames can be combined into a batch to improve processing efficiency.

[0138] The present application obtains the corresponding first video frame by resizing the video frame; then crops the first video frame to obtain the corresponding second video frame; then normalizes the second video frame to obtain the corresponding third video frame; subsequently performs format conversion on the third video frame to obtain the corresponding fourth video frame; and finally uses the fourth video frame as the designated video frame. The present application can intelligently and accurately complete the preprocessing of the video frame by resizing, cropping, normalizing, and format conversion on the video frame, thereby effectively ensuring that the data format of the designated video frame obtained is consistent with the model input requirements.

[0139] In some optional implementations of this embodiment, step S205 includes the following steps:

[0140] The high-resolution video frame is subjected to denoising processing to obtain a corresponding first designated video frame.

[0141] In this embodiment, the high-resolution video frame may be denoised by using an image processing algorithm to reduce noise and artifacts.

[0142] The first designated video frame is subjected to contrast enhancement processing to obtain a corresponding second designated video frame.

[0143] In this embodiment, the contrast of the video frame can be adjusted to make it clearer and easier to observe.

[0144] Perform format conversion processing on the second designated video frame to obtain a corresponding third designated video frame.

[0145] In this embodiment, the video frame can be converted from the model output format back to the original format (such as from RGB back to BGR).

[0146] The third designated video frame is resized to obtain a corresponding fourth designated video frame.

[0147] In this embodiment, if the video frame size is previously reduced for model input, it can be restored to the original size by using an interpolation algorithm.

[0148] The fourth designated video frame is used as the target video frame.

[0149] The present application performs denoising on the high-resolution video frame to obtain the corresponding first designated video frame; then performs contrast enhancement on the first designated video frame to obtain the corresponding second designated video frame; then performs format conversion on the second designated video frame to obtain the corresponding third designated video frame; subsequently performs size restoration on the third designated video frame to obtain the corresponding fourth designated video frame; and finally uses the fourth designated video frame as the target video frame. The present application performs denoising, contrast enhancement, format conversion, and size restoration on the high-resolution video frame, thereby intelligently and accurately completing the image quality optimization processing for the high-resolution video frame, and effectively improving the quality of the generated target video frame.

[0150] In some optional implementations of this embodiment, after step S206, the electronic device may further perform the following steps:

[0151] Call up the preset video file library.

[0152] In this embodiment, for different video sources, video files for storing video frames of the video sources are pre-constructed and stored in the video file library.

[0153] A target video file corresponding to the video source is obtained from the video file library.

[0154] In this embodiment, the target video file corresponding to the video source may be screened out from the video file library according to the video source type of the video source.

[0155] The target video frame is stored in the target video file.

[0156] In this embodiment, the target video frames may be saved frame by frame into the target video file, thereby forming a complete optimized video.

[0157] The present application calls a preset video file library; then obtains the target video file corresponding to the video source from the video file library; and subsequently stores the target video frame in the target video file. The present application obtains the target video file corresponding to the video source from the called video file library, and then stores the target video frame in the target video file, thereby ensuring the storage standardization of the obtained target video frame, and using the target video file to store the target video frame can be conducive to forming a complete optimized video, thereby improving the storage intelligence of the target video frame.

[0158] In some optional implementations, the user information obtained is subject to the user's consent and complies with relevant laws and policies.

[0159] In addition, the non-Company software tools or components that appear in the embodiments of the present application are merely examples and do not represent actual use.

[0160] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0161] It should be emphasized that in order to further ensure the privacy and security of the above-mentioned target video frame, the above-mentioned target video frame can also be stored in a blockchain node.

[0162] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0163] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0164] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0165] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0166] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0167] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of a video frame processing device, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0168] like Figure 3 As shown, the video frame processing device 300 described in this embodiment includes: a first acquisition module 301, a pre-processing module 302, a first processing module 303, a second processing module 304 and a display module 305. Among them:

[0169] A first acquisition module 301 is used to acquire a video frame from a preset video source;

[0170] A preprocessing module 302, used to preprocess the video frame to obtain a corresponding designated video frame;

[0171] The first processing module 303 is used to perform super-resolution processing on the specified video frame based on a preset target image processing model to generate a corresponding high-resolution video frame; wherein the target image processing model is a model generated by training the optimized specified diffusion model based on a pre-constructed super-resolution data set;

[0172] The second processing module 304 is used to perform image quality optimization processing on the high-resolution video frame to obtain a corresponding target video frame;

[0173] The display module 305 is used to display the target video frame on a preset monitoring screen.

[0174] In this embodiment, the operations performed by the above modules or units correspond one-to-one to the steps of the video frame processing method in the aforementioned implementation mode, and are not described in detail here.

[0175] In some optional implementations of this embodiment, the video frame processing device further includes:

[0176] The first calling module is used to call the initial pre-trained diffusion model;

[0177] A second acquisition module is used to acquire a specified network structure of the pre-trained diffusion model;

[0178] An optimization module, used to freeze the encoder layer of the specified network structure based on a preset freezing strategy, and to fine-tune the decoder part of the specified network structure based on a preset fine-tuning strategy to obtain a corresponding optimization model;

[0179] The first determining module is used to use the optimization model as the designated diffusion model.

[0180] In this embodiment, the operations performed by the above modules or units correspond one-to-one to the steps of the video frame processing method in the aforementioned implementation mode, and are not described in detail here.

[0181] In some optional implementations of this embodiment, the video frame processing device further includes:

[0182] A third acquisition module is used to acquire a pre-constructed super-resolution dataset; wherein the super-resolution dataset includes image pairs with corresponding relationships between low resolution and high resolution;

[0183] A division module, used for dividing the super-resolution dataset into a training set and a validation set;

[0184] A second determination module, used to determine an overall loss function corresponding to the specified diffusion model;

[0185] A training module, configured to train the specified diffusion model using the training set based on the overall loss function and a preset optimization algorithm to obtain a trained first diffusion model;

[0186] A third processing module, configured to evaluate and optimize the first diffusion model based on the verification set to obtain a second diffusion model that meets preset performance requirements;

[0187] The third determining module is configured to use the second diffusion model as the target image processing model.

[0188] In this embodiment, the operations performed by the above modules or units correspond one-to-one to the steps of the video frame processing method in the aforementioned implementation mode, and are not described in detail here.

[0189] In some optional implementations of this embodiment, the third processing module includes:

[0190] An evaluation submodule, configured to evaluate the first diffusion model based on the validation set to obtain a corresponding evaluation index score;

[0191] An analysis submodule is used to analyze and process the evaluation index scores to obtain corresponding analysis results;

[0192] An adjustment submodule, configured to adjust the training strategy of the specified diffusion model based on the analysis result to obtain an adjusted target training strategy;

[0193] A training submodule, configured to retrain the first diffusion model based on the target training strategy, and repeat the evaluation and optimization process until a specified model that meets the performance requirements is obtained;

[0194] The first determining submodule is configured to use the designated model as the second diffusion model.

[0195] In this embodiment, the operations performed by the above modules or units correspond one-to-one to the steps of the video frame processing method in the aforementioned implementation mode, and are not described in detail here.

[0196] In some optional implementations of this embodiment, the preprocessing module 302 includes:

[0197] A first processing submodule, configured to perform a size adjustment process on the video frame to obtain a corresponding first video frame;

[0198] A second processing submodule, configured to perform cropping processing on the first video frame to obtain a corresponding second video frame;

[0199] A third processing submodule, configured to perform normalization processing on the second video frame to obtain a corresponding third video frame;

[0200] A fourth processing submodule, configured to perform format conversion processing on the third video frame to obtain a corresponding fourth video frame;

[0201] The second determining submodule is configured to use the fourth video frame as the designated video frame.

[0202] In this embodiment, the operations performed by the above modules or units correspond one-to-one to the steps of the video frame processing method in the aforementioned implementation mode, and are not described in detail here.

[0203] In some optional implementations of this embodiment, the second processing module 304 includes:

[0204] A fifth processing submodule, configured to perform denoising on the high-resolution video frame to obtain a corresponding first designated video frame;

[0205] a sixth processing submodule, configured to perform contrast enhancement processing on the first designated video frame to obtain a corresponding second designated video frame;

[0206] a seventh processing submodule, configured to perform format conversion processing on the second designated video frame to obtain a corresponding third designated video frame;

[0207] an eighth processing submodule, configured to perform size restoration processing on the third designated video frame to obtain a corresponding fourth designated video frame;

[0208] The third determining submodule is configured to use the fourth designated video frame as the target video frame.

[0209] In this embodiment, the operations performed by the above modules or units correspond one-to-one to the steps of the video frame processing method in the aforementioned implementation mode, and are not described in detail here.

[0210] In some optional implementations of this embodiment, the video frame processing device further includes:

[0211] The second calling module is used to call a preset video file library;

[0212] A fourth acquisition module, used to acquire a target video file corresponding to the video source from the video file library;

[0213] The storage module is used to store the target video frame into the target video file.

[0214] In this embodiment, the operations performed by the above modules or units correspond one-to-one to the steps of the video frame processing method in the aforementioned implementation mode, and are not described in detail here.

[0215] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0216] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 4 with components 41-43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0217] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0218] The memory 41 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. equipped on the computer device 4. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions of the video frame processing method, etc. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.

[0219] The processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run computer-readable instructions stored in the memory 41 or process data, such as computer-readable instructions for running the video frame processing method.

[0220] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0221] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0222] In an embodiment of the present application, the present application first obtains a video frame from a preset video source; then preprocesses the video frame to obtain a corresponding designated video frame; then performs super-resolution processing on the designated video frame based on a preset target image processing model to generate a corresponding high-resolution video frame; subsequently performs image quality optimization processing on the high-resolution video frame to obtain a corresponding target video frame; and finally displays the target video frame on a preset monitoring screen. The present application obtains a designated video frame by preprocessing the video frame obtained from the video source, and then performs super-resolution processing on the designated video frame based on a target image processing model generated by training the optimized designated diffusion model based on a super-resolution data set to generate a corresponding high-resolution video frame, and further performs image quality optimization processing on the high-resolution video frame to obtain a corresponding target video frame, thereby achieving efficient and accurate improvement of the resolution of the video frame, which is beneficial to improving the quality of video imaging.

[0223] The present application also provides another implementation, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the video frame processing method as described above.

[0224] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0225] In an embodiment of the present application, the present application first obtains a video frame from a preset video source; then preprocesses the video frame to obtain a corresponding designated video frame; then performs super-resolution processing on the designated video frame based on a preset target image processing model to generate a corresponding high-resolution video frame; subsequently performs image quality optimization processing on the high-resolution video frame to obtain a corresponding target video frame; and finally displays the target video frame on a preset monitoring screen. The present application obtains a designated video frame by preprocessing the video frame obtained from the video source, and then performs super-resolution processing on the designated video frame based on a target image processing model generated by training the optimized designated diffusion model based on a super-resolution data set to generate a corresponding high-resolution video frame, and further performs image quality optimization processing on the high-resolution video frame to obtain a corresponding target video frame, thereby achieving efficient and accurate improvement of the resolution of the video frame, which is beneficial to improving the quality of video imaging.

[0226] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0227] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A video frame processing method, characterized in that: The steps include: Get video frames from a preset video source; Preprocessing the video frame to obtain a corresponding designated video frame; Performing super-resolution processing on the designated video frame based on a preset target image processing model to generate a corresponding high-resolution video frame; wherein the target image processing model is a model generated by training an optimized designated diffusion model based on a pre-constructed super-resolution data set; Performing image quality optimization processing on the high-resolution video frame to obtain a corresponding target video frame; The target video frame is displayed on a preset monitoring screen.

2. The video frame processing method according to claim 1, characterized in that: The step of performing super-resolution processing on the designated video frame based on the preset target image processing model to generate a corresponding high-resolution video frame further includes: Call the preset pre-trained diffusion model; Obtaining a specified network structure of the pre-trained diffusion model; Freezing the encoder layer of the specified network structure based on a preset freezing strategy, and fine-tuning the decoder part of the specified network structure based on a preset fine-tuning strategy to obtain a corresponding optimization model; The optimized model is used as the specified diffusion model.

3. The video frame processing method according to claim 1, characterized in that: The step of performing super-resolution processing on the designated video frame based on the preset target image processing model to generate a corresponding high-resolution video frame further includes: Acquire a pre-constructed super-resolution dataset; wherein the super-resolution dataset includes image pairs with corresponding relationships between low resolution and high resolution; Dividing the super-resolution dataset into a training set and a validation set; determining an overall loss function corresponding to the specified diffusion model; Based on the overall loss function and a preset optimization algorithm, the designated diffusion model is trained using the training set to obtain a trained first diffusion model; Evaluate and optimize the first diffusion model based on the validation set to obtain a second diffusion model that meets preset performance requirements; The second diffusion model is used as the target image processing model.

4. The video frame processing method according to claim 3, characterized in that: The step of evaluating and optimizing the first diffusion model based on the validation set to obtain a second diffusion model that meets the preset performance requirements specifically includes: Performing evaluation processing on the first diffusion model based on the validation set to obtain a corresponding evaluation index score; Analyze and process the evaluation index scores to obtain corresponding analysis results; Adjusting the training strategy of the specified diffusion model based on the analysis result to obtain an adjusted target training strategy; Retraining the first diffusion model based on the target training strategy, and repeating the evaluation and optimization process until a specified model that meets the performance requirements is obtained; The designated model is used as the second diffusion model.

5. The video frame processing method according to claim 1, characterized in that: The step of preprocessing the video frame to obtain the corresponding designated video frame specifically includes: Resizing the video frame to obtain a corresponding first video frame; Performing cropping processing on the first video frame to obtain a corresponding second video frame; Normalizing the second video frame to obtain a corresponding third video frame; Performing format conversion processing on the third video frame to obtain a corresponding fourth video frame; The fourth video frame is used as the designated video frame.

6. The video frame processing method according to claim 1, characterized in that: The step of performing image quality optimization processing on the high-resolution video frame to obtain a corresponding target video frame specifically includes: Performing denoising processing on the high-resolution video frame to obtain a corresponding first designated video frame; Performing contrast enhancement processing on the first designated video frame to obtain a corresponding second designated video frame; Performing format conversion processing on the second designated video frame to obtain a corresponding third designated video frame; Performing size restoration processing on the third designated video frame to obtain a corresponding fourth designated video frame; The fourth designated video frame is used as the target video frame.

7. The video frame processing method according to claim 1, characterized in that: After the step of performing image quality optimization processing on the high-resolution video frame to obtain a corresponding target video frame, the method further includes: Call the preset video file library; Acquire a target video file corresponding to the video source from the video file library; The target video frame is stored in the target video file.

8. A video frame processing device, characterized in that: include: A first acquisition module, used for acquiring video frames from a preset video source; A preprocessing module, used for preprocessing the video frame to obtain a corresponding designated video frame; A first processing module is used to perform super-resolution processing on the specified video frame based on a preset target image processing model to generate a corresponding high-resolution video frame; wherein the target image processing model is a model generated by training an optimized specified diffusion model based on a pre-constructed super-resolution data set; A second processing module is used to perform image quality optimization processing on the high-resolution video frame to obtain a corresponding target video frame; The display module is used to display the target video frame on a preset monitoring screen.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the video frame processing method according to any one of claims 1 to 7 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the video frame processing method according to any one of claims 1 to 7 are implemented.

Images