Display for image processing, image processing system and image processing method

Abstract

The invention belongs to the technical field of displays, and particularly discloses a display for image processing, an image processing system and an image processing method. The display comprises an instruction input interface, an image recognition module, an image amplification processing module and a display panel, the instruction input interface is respectively connected with the image identification module and the image amplification processing module; the instruction input interface is used for receiving an externally input image amplification instruction; the image magnification instruction carries position information of a to-be-processed area of a to-be-processed image displayed by the display panel; the image recognition module is used for recognizing the content type of a target image block corresponding to the region position information; and the image magnification processing module is used for performing magnification processing on the target image block according to a magnification mode corresponding to the content type in response to the image magnification instruction, and sending the processed image information to the display panel for display. According to the method and the device, limitation of external software and platforms can be avoided, and efficient processing of the display image of the display can be conveniently realized.

Description

Technical Field

[0001] This application belongs to the technical field of displays, and more specifically, relates to a display for image processing, an image processing system, and an image processing method. Background Art

[0002] Currently, for the processing of images displayed on a display, such as image magnification processing, traditional processing methods generally rely on image processing software installed on an external system.

[0003] However, in practical applications, some image processing software cannot be fully compatible with different system platforms, such as embedded platforms. Due to the incompatibility between the image processing software and the system platform, the system cannot process the images displayed on the display, which brings a lot of inconvenience to some users and results in a poor user experience. Summary of the Invention

[0004] Aiming at the deficiencies of the prior art, this application aims to solve the problem that in the prior art, due to the incompatibility between the image processing software and the system platform, the system cannot process the images displayed on the display.

[0005] To achieve the above object, in a first aspect, this application provides a display for image processing, including: An instruction input interface, an image recognition module, an image magnification processing module, and a display panel; The instruction input interface is respectively connected to the image recognition module and the image magnification processing module; the image magnification processing module is respectively connected to the image recognition module and the display panel; The instruction input interface is used to receive an externally input image magnification instruction; the image magnification instruction carries the area position information of the image to be processed displayed on the display panel; The image recognition module is used to identify the content type of the target image block corresponding to the area position information; The image magnification processing module is used to respond to the image magnification instruction, magnify the target image block according to the magnification method corresponding to the content type, and send the processed image information to the display panel for display.

[0006] Optionally, the content type includes a text type and a non-text type; The image magnification processing module is used to, when determining that the content type is the text type, magnify the text information included in the target image block by using a local pixel processing and compensation mechanism; Or, The image magnification processing module is used to perform image magnification processing on the target image block according to the magnification method corresponding to the image magnification factor when it is determined that the content type is non-text type.

[0007] Optionally, the image magnification processing module is used to perform image magnification processing on the target image block by using a first image magnification method when it is determined that the content type is non-text type and the image magnification factor is less than the target magnification threshold; Or, The image magnification processing module is used to perform image magnification processing on the target image block by using a second image magnification method when it is determined that the content type is non-text type and the image magnification factor is not less than the target magnification threshold.

[0008] Optionally, the display further includes a video image enhancement module; The video image enhancement module is connected to the instruction input interface; The instruction input interface is further used to receive an externally input video image enhancement instruction; The video image enhancement module is used to perform video image enhancement processing on the to-be-processed video file with a first resolution specified by the video image enhancement instruction, and generate a target video file with a second resolution.

[0009] Optionally, the video image enhancement module includes a preprocessing module, an image enhancement module, a timing processing module, an image denoising module, and an image synthesis module that are connected in sequence; The preprocessing module is used to perform video decoding on the to-be-processed video file to obtain each frame of video image included in the to-be-processed video file, and perform frame alignment on each frame of the video image to obtain the aligned frames of video images; The image enhancement module is used to extract the residual images corresponding to each frame of the aligned video images, and generate each frame of target video images with the second resolution based on each frame of the aligned video images and the corresponding residual images; The timing processing module is used to perform timing smoothing processing on each frame of the target video images; The image denoising module is used to perform denoising and detail enhancement on each frame of the video images output by the timing processing module to obtain each frame of enhanced target video images; The image synthesis module is used to synthesize each frame of the enhanced target video images to generate the target video file with the second resolution.

[0010] Optionally, the display further includes: an image input interface; The image input interface is connected to an external image source, and is used to receive an image to be processed from the external image source, and transmit the image to be processed to the display panel for display; and is used to receive a video file to be processed from the external image source.

[0011] In a second aspect, the present application provides an image processing system, comprising a command input device and an image source, and also comprising any of the aforementioned displays; The command input device is connected to the image source and the command input interface of the display respectively; the display panel of the display is connected to the image source; The command input device is used to control the image source to transmit the image to be processed to the display panel of the display for display; The command input device is also used for a user to input an image magnification command for the image to be processed, and transmit the image magnification command to the command input interface of the display; the image magnification command carries the location information of the area to be processed of the image to be processed; The image recognition module of the display is used to identify the content type of the target image block corresponding to the area position information; The image magnification processing module of the display is used to respond to the image magnification instruction, magnify the target image block according to the magnification method corresponding to the content type, and send the processed image information to the display panel for display.

[0012] In a third aspect, the present application provides an image processing method applied to any of the aforementioned displays, comprising: receiving an externally input image magnification instruction; the image magnification instruction carries the location information of the area to be processed of the image to be processed displayed by the display panel; Identifying a content type of a target image block corresponding to the region position information; In response to the image magnification instruction, the target image block is magnified according to the magnification method corresponding to the content type, and the processed image information is displayed.

[0013] Optionally, the content type includes a text type and a non-text type; and in response to the image magnification instruction, performing magnification processing on the target image block according to a magnification method corresponding to the content type includes: In response to the image magnification instruction, when it is determined that the content type is a text type, amplifying the text information contained in the target image block by using a local pixel processing and compensation mechanism; or, In response to the image magnification instruction, when it is determined that the content type is non-text type, perform image magnification processing on the target image block according to the magnification method corresponding to the image magnification factor.

[0014] Optionally, the performing image magnification processing on the target image block according to the magnification method corresponding to the image magnification factor in response to the image magnification instruction when it is determined that the content type is non-text type includes: In response to the image magnification instruction, when it is determined that the content type is non-text type and the image magnification factor is less than the target magnification threshold, perform image magnification processing on the target image block using the first image magnification method; Or, In response to the image magnification instruction, when it is determined that the content type is non-text type and the image magnification factor is not less than the target magnification threshold, perform image magnification processing on the target image block using the second image magnification method.

[0015] Generally speaking, compared with the prior art, the above technical solution conceived by the present application has the following beneficial effects: A display, an image processing system, and an image processing method for image processing provided by the present application optimize the internal hardware structure of the display by introducing an instruction input interface, an image recognition module, and an image magnification processing module, enabling the display to have the function of image processing locally. Thus, there is no need to consider the compatibility problem between traditional image processing software and the system platform. At the same time, by identifying the content type of the image displayed on the display and specifically calling the matching magnification method for image magnification processing, it is possible to effectively eliminate the dependence on external image processing software and platforms, and at the same time, it is possible to conveniently achieve efficient processing of the image displayed on the display and improve the user experience. Description of the Drawings

[0016] Figure 1 is one of the schematic structural diagrams of the display for image processing provided by the embodiments of the present application; Figure 2 is another schematic structural diagram of the display for image processing provided by the embodiments of the present application; Figure 3 is yet another schematic structural diagram of the display for image processing provided by the embodiments of the present application; Figure 4 is the schematic flowchart of the image processing method provided by the embodiments of the present application; Figure 5 is the schematic structural diagram of the image processing system provided by the embodiments of the present application. Detailed Embodiments

[0017] To make the objectives, technical solutions and advantages of this application more clear and understandable, the following further elaborates on this application in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not used to limit this application.

[0018] The terms "first" and "second" etc. in the description and claims of this application are used to distinguish different objects, rather than to describe a specific order of the objects. For example, the first resolution and the second resolution etc. are used to distinguish different resolutions, rather than to describe a specific order of the resolutions.

[0019] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design solution described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Exactly speaking, using words such as "exemplary" or "for example" aims to present relevant concepts in a specific manner.

[0020] In the description of the embodiments of this application, unless otherwise specified, the meaning of "a plurality of" refers to two or more. For example, a plurality of computer instructions refers to two or more computer instructions etc.

[0021] The following describes the embodiments of this application in conjunction with the accompanying drawings in the embodiments of this application.

[0022] Figure 1 is one of the schematic structural diagrams of a display for image processing provided by the embodiments of this application. As Figure 1 shown, it includes: an instruction input interface 101, an image recognition module 102, an image magnification processing module 103, and a display panel 104; The instruction input interface 101 is respectively connected to the image recognition module 102 and the image magnification processing module 103; the image magnification processing module 103 is respectively connected to the image recognition module 102 and the display panel 104; The instruction input interface 101 is used to receive an externally input image magnification instruction; the image magnification instruction carries the area position information of the area to be processed of the image to be processed displayed on the display panel. The image recognition module 102 is used to identify the content type of the target image block corresponding to the area position information. The image magnification processing module 103 is used to respond to the image magnification instruction, magnify the target image block according to the magnification method corresponding to the content type, and send the processed image information to the display panel 104 for display.

[0023] Specifically, the target image block described in the embodiments of the present application refers to the image block information composed of all pixel points covered by the image area to be processed in the image to be processed.

[0024] The content types described in the embodiments of the present application may include text types and non-text types. Among them, the non-text type means that the main content information displayed by the target image block does not contain a large amount of text information. The non-text type of images may specifically include portrait images, landscape images, etc.

[0025] In the embodiments of the present application, the internal hardware structure of the display is designed by using an instruction input interface, an image recognition module, an image magnification processing module, and a display panel, so that the display can locally implement an image processing function. Among them, the instruction input interface is respectively connected to the image recognition module and the image magnification processing module, and the image magnification processing module is respectively connected to the image recognition module and the display panel.

[0026] It should be noted that the display in the embodiments of the present application may be a display with a touch screen operation function or a display without a touch screen operation function.

[0027] In the embodiments of the present application, the display is designed with an image magnification mode. In a scenario where the display does not have a touch screen operation function, it can be connected to an external input expansion device, such as a mouse, a keyboard, etc., through the instruction input interface set on the display. Here, the instruction input interface may specifically adopt a USB interface. Thus, the user can control the display with the help of the external input expansion device and make the display enter the image magnification mode. In the image magnification mode, the user can perform targeted input operations on the image to be processed displayed on the display panel of the display.

[0028] It should be noted that the image to be processed displayed on the display panel of the display may be an image pre-stored locally on the display or an image transmitted from an external image source. The present application does not make specific limitations on this.

[0029] After the front-end user inputs an image magnification instruction operation, the instruction input interface of the display can receive the externally input image magnification instruction. It can be understood that through the user's box selection input operation, the image magnification instruction can carry the position information of the area to be processed of the image to be processed displayed on the display panel, and at the same time, it can also carry the required image magnification factor.

[0030] Optionally, in the embodiments of the present application, in a scenario where the display has a touch screen operation function, the input of the image magnification instruction can also be realized through the user's touch screen operation on the display panel.

[0031] Further, in the embodiments of the present application, the image recognition module inside the display can identify the target image block corresponding to the area position information and the content type corresponding to the image information according to the area position information of the image to be processed carried by the image magnification instruction.

[0032] In the embodiments of the present application, the image recognition module further sends the recognized content type to the image magnification processing module, and the image magnification processing module can call the corresponding image processing algorithm according to the content type corresponding to the target image block. In this way, after receiving the image magnification instruction and the content type corresponding to the target image block, the image magnification processing module can respond to the image magnification instruction, magnify the target image block according to the magnification method corresponding to the content type, and send the processed image information to the display panel for display.

[0033] In the embodiments of the present application, after the display exits the image magnification mode, the display can also generate a corresponding picture through the computer vision library OPENCV for the image pixels after area magnification processing and save it locally for the user to use next time.

[0034] The display for image processing in the embodiments of the present application optimizes the internal hardware structure of the display by introducing an instruction input interface, an image recognition module, and an image magnification processing module, enabling the display to have the function of image processing locally. Thus, there is no need to consider the compatibility issues between traditional image processing software and system platforms. At the same time, by identifying the content type of the image displayed on the display and specifically calling the matching magnification method for image magnification processing, it can effectively eliminate the dependence on external image processing software and platforms. Meanwhile, it can conveniently achieve the efficient processing of the image displayed on the display and improve the user experience.

[0035] Figure 2 is the second structural schematic diagram of the display for image processing provided by the embodiments of the present application, as Figure 2 shown. As an optional embodiment, the display further includes: an image input interface 105; The image input interface 105 is connected to an external image source, and is used to receive the image to be processed transmitted from the external image source and transmit the image to be processed to the display panel 104 for display; and is used to receive the video file to be processed transmitted from the external image source.

[0036] Specifically, in the embodiments of the present application, the display further includes an image input interface. Through the image input interface, the display can be directly connected to an external image source. In this way, the display can receive the image to be processed that the user needs to process transmitted from the external image source and transmit the image to the display panel for display.

[0037] In addition, an external image source can also transmit video files. Through the image input interface, the display can receive the to-be-processed video files that the user needs to process from the external image source. The display can further store the video files or display them to the user through the display panel.

[0038] The display for image processing according to the embodiments of the present application realizes the communication connection between the display and the external image source by introducing the image input interface, facilitating the display to process different image processing requirements of different users and improving the image processing performance of the display.

[0039] Based on the content of the above embodiments, as an optional embodiment, the content type includes text type and non-text type; The image magnification processing module is used to magnify the text information included in the target image block by using local pixel processing and compensation mechanism when it is determined that the content type is text type; Or, The image magnification processing module is used to perform image magnification processing on the target image block according to the magnification method corresponding to the image magnification factor when it is determined that the content type is non-text type.

[0040] Specifically, in the embodiments of the present application, the image magnification processing module of the display can perform image magnification processing in different ways according to different content types of the target image block.

[0041] Specifically, on the one hand, when the image magnification processing module determines that the content type of the target image block is a text type with low requirements for image quality, it can call the local pixel processing and compensation mechanism to quickly magnify the text information included in the target image block. The local pixel processing algorithm can identify the text area in the target image block, highlight the text details, and enhance the image resolution through steps such as local area recognition, pixel interpolation, and local enhancement; the compensation mechanism can improve the overall quality of the image through edge compensation, color compensation, and detail compensation. Overall, it can improve the readability and clarity of the text area in the target image block without significantly increasing the overall size of the image.

[0042] On the other hand, when the image magnification processing module determines that the content type of the target image block is non-text type, it will further read the image magnification factor carried by the image magnification instruction and perform image magnification processing on the target image block by using different magnification methods according to different image magnification factors.

[0043] Optionally, in the embodiments of the present application, for the scenario of non-text type images that require high-quality magnification, according to different image magnification factors, an interpolation algorithm or a transform-based algorithm can be selected.

[0044] The display device according to the embodiment of the present application can significantly improve the image quality, retain details, and improve the image processing efficiency and further enhance the performance of the display device in image processing by designing various processing methods of the image magnification processing module and selecting different image magnification methods for magnification processing according to different image contents.

[0045] Based on the content of the above embodiment, as an optional embodiment, the image magnification processing module is configured to perform image magnification processing on a target image block by using a first image magnification method when it is determined that the content type is a non-text type and the image magnification factor is less than a target magnification threshold; Or, The image magnification processing module is configured to perform image magnification processing on the target image block by using a second image magnification method when it is determined that the content type is a non-text type and the image magnification factor is not less than the target magnification threshold.

[0046] Specifically, the target magnification threshold described in the embodiment of the present application is used to define the magnification degree of the image magnification factor, and it can be specifically set according to the actual processing requirements of the user. When the image magnification factor is less than the target magnification threshold, it can be considered a small-scale magnification operation, and when the image magnification factor is not less than the target magnification threshold, it can be considered a medium or large-scale magnification operation.

[0047] In the embodiment of the present application, when the image magnification processing module of the display device determines that the content type of the target image block is a non-text type, it will further obtain the image magnification factor carried in the image magnification instruction. When it is determined that the image magnification factor is less than the target magnification threshold, it indicates that a small-scale magnification of the target image block is required. To ensure the image magnification effect and efficiency, the first image magnification method can be used for small-scale magnification processing at this time. For example, the first image magnification method can specifically adopt the bilinear interpolation method to smoothly process the transition between pixels when magnifying the image, calculate the value of the new pixel by weighted averaging of four adjacent pixels to adapt to the magnified position, so as to achieve fast magnification processing of the target image block.

[0048] In an embodiment of the present application, when the image magnification processing module of the display obtains the image magnification factor carried in the image magnification instruction and determines that the image magnification factor is not less than the target magnification threshold, it indicates that medium or large-scale magnification of the target image block is required. To ensure the quality of the magnified image, the second image magnification method can be used for corresponding magnification processing at this time. For example, the second image magnification method can specifically adopt the bicubic interpolation method, which considers 16 adjacent pixel points around the target pixel point, and the interpolation process is of a high order and uses a more complex weighting function for processing, capable of generating a relatively smooth image scaling result, reducing the jagged phenomenon and blurring effect. At the same time, it can better preserve the details and texture information of the image, making the magnified image look more natural and real.

[0049] For the display in the embodiment of the present application, for different magnitudes of image magnification requirements, by adopting different image magnification algorithms for targeted processing, it can achieve an adaptive image processing effect according to different magnification requirements, optimize the quality and efficiency of image magnification, and further improve the performance of display image processing.

[0050] In an embodiment of the present application, the display can also be built-in with a video image enhancement function, which is mainly achieved by introducing a video image enhancement module.

[0051] Figure 3 This is the third schematic diagram of the structure of the display for image processing provided by the embodiment of the present application. As Figure 3 shown, as an optional embodiment, the display further includes a video image enhancement module 106; The video image enhancement module 106 is connected to the instruction input interface 101; The instruction input interface 101 is further configured to receive an externally input video image enhancement instruction; The video image enhancement module 106 is configured to perform video image enhancement processing on the to-be-processed video file with the first resolution specified by the video image enhancement instruction in response to the video image enhancement instruction, and generate a target video file with the second resolution.

[0052] Specifically, the first resolution described in the embodiment of the present application refers to the original resolution of the to-be-processed video file, which is used to represent a video resolution relatively lower than the second resolution. It can be understood that the second resolution represents a relatively higher video resolution. For example, the first resolution is a 2K video resolution, and the second resolution is a 4K video resolution.

[0053] In an embodiment of the present application, by connecting a video image enhancement module to an instruction input interface, after a front-end user inputs a video image enhancement instruction through an external input device, the instruction input interface can receive the externally input video image enhancement instruction and forward the video image enhancement instruction to the video image enhancement module. Furthermore, the video image enhancement module will, in response to the video image enhancement instruction, perform video image enhancement processing on the to-be-processed video file with the first resolution specified by the video image enhancement instruction to generate a target video file with the second resolution.

[0054] It should be noted that the to-be-processed video file can be a video file pre-stored locally on the display or a video file transmitted from an external video image source. The present application does not make specific limitations in this regard.

[0055] The display in the embodiment of the present application, by introducing a video image enhancement module and connecting it to the instruction input interface, ensures the information interaction between the video image enhancement module and external instructions, and effectively realizes the video image enhancement function of the display.

[0056] In the actual specific implementation process, to solve the above problems of the video image enhancement function, it is mainly divided into three aspects of design, including data preparation preprocessing, super-resolution processing, and quality evaluation and optimization.

[0057] Based on the content of the above embodiment, as an alternative embodiment, the video image enhancement module includes a preprocessing module, an image enhancement module, a timing processing module, an image denoising module, and an image synthesis module connected in sequence; The preprocessing module is used to perform video decoding on the to-be-processed video file to obtain each frame of video image included in the to-be-processed video file, and perform frame alignment on each frame of video image to obtain each frame of aligned video image; The image enhancement module is used to extract the residual images corresponding to each frame of aligned video image, and generate each frame of target video image with the second resolution based on each frame of aligned video image and the corresponding residual images; The timing processing module is used to perform timing smoothing processing on each frame of target video image; The image denoising module is used to perform denoising and detail enhancement on each frame of video image output by the timing processing module to obtain each frame of enhanced target video image; The image synthesis module is used to synthesize each frame of enhanced target video image to generate a target video file with the second resolution.

[0058] Specifically, in the embodiments of the present application, the video image enhancement module includes a preprocessing module, an image enhancement module, a timing processing module, an image denoising module, and an image synthesis module connected in sequence. Among them, the image enhancement module can be constructed using a super-resolution (Very Deep Convolutional Networks SR, VDSR) model based on deep learning, and the super-resolution performance of the image is improved through a deeper network structure.

[0059] Among them, the data preparation preprocessing stage is completed by the preprocessing module, and its specific implementation method can include the following steps: Video decoding: Decode a video file with the first resolution, such as a 2K video, into frames of images for subsequent processing; Frame alignment processing: When there are slight movements or distortions in the video frames, the optical flow estimation method of motion compensation technology can be used for frame alignment to reduce the blurring phenomenon caused by frame misalignment.

[0060] Through the above processing of the preprocessing module, the aligned video images of each frame corresponding to the video file to be processed can be obtained.

[0061] Furthermore, the super-resolution processing stage is completed through the cooperation of the image enhancement module, the timing processing module, the image denoising module, and the image synthesis module.

[0062] Among them, through the image enhancement module, the trained VDSR model network is used to perform image enhancement processing on each frame of the video image. Specifically, the VDSR model network extracts the residual image corresponding to each frame of the aligned video image according to the features in the low-resolution image, and superimposes the aligned video images of each frame and the corresponding residual images to generate the target video images of each frame with a 4K resolution, that is, the corresponding high-resolution images are obtained. This process is performed in each frame of the video to restore the details and textures of the image.

[0063] Among them, the VDSR model can adopt the L2 loss function, also known as the Mean Squared Error (MSE), which calculates the pixel-level difference between the reconstructed image and the real high-resolution image. Specifically, the loss function can be expressed as: ; In the formula, represents the i th pixel value of the super-resolution image predicted by the model, represents the i th pixel value of the real high-resolution image, N represents the total number of pixels in the image.

[0064] In the embodiments of the present application, the following four optimizations can be made for the relatively high computational complexity and large amount of training data of the VDSR model: First, adopt the strategy of residual learning, which not only helps to accelerate the training of the network, but also can reduce the computational cost. Specifically, the network of VDSR learns the residual between the low-resolution image and the high-resolution image, rather than directly generating the high-resolution image from the low-resolution image. In this way, the network can focus on learning the subtle differences, rather than having to learn the mapping of the entire image. This approach has the following characteristics: Accelerated convergence: Through residual learning, VDSR can converge to a better solution relatively quickly, thereby reducing the number of iterations and computational overhead during training; Reduced parameter redundancy: The network only needs to learn the difference between the input and the output. Compared with directly learning the mapping from low resolution to high resolution, the number of parameters and computational complexity are effectively reduced.

[0065] Second, set a smaller convolution kernel size: The VDSR model can use a small 3x3 convolution kernel instead of a larger one. Smaller convolution kernels (such as 3x3) have significantly less computational overhead than larger ones (such as 5x5 or 7x7), and since convolution operations are local, they can capture local features of the image more effectively. This approach has the following characteristics: Reduced computational amount: The 3x3 convolution kernel has less computational amount per layer than larger-sized convolution kernels, especially in deep networks, reducing the overall computational burden; Maintain feature expression ability: Even small convolution kernels can effectively extract complex features of the image after being stacked in multiple layers, so the performance ability is not sacrificed.

[0066] Third, adopt skip connections: Transmit the residual information between the low-resolution input image and the high-resolution output image. This method reduces the complex transformation of features in each layer, reduces the content that the network needs to learn, and can transmit information more efficiently. This approach has the following advantages: More efficient information transmission: Skip connections enable the network to directly transmit the original information of the input image, avoiding excessive complex transformations in multiple layers of the network and saving computational resources; Reduce redundant learning: This method reduces redundant learning steps, especially in deep networks, reducing the computational amount.

[0067] Fourth, the VDSR model can adopt a relatively simple convolution layer design to avoid complex network structures, such as fully connected layers or excessive non-linear activation function layers. This design reduces the computational complexity of each layer. Through techniques such as weight sharing, the number of network parameters and computational amount are reduced. This approach has the following advantages: Reducing the number of parameters: The convolutional layer reduces the number of parameters to be optimized during the training process by sharing weights, thus reducing the computational burden; Computational optimization: By optimizing the convolutional layer structure, the computational efficiency can be improved. Especially through parallel processing of convolutional operations, the computational cost can be significantly reduced.

[0068] The VDSR model itself already uses a relatively deep network for super-resolution processing. However, in practical applications, the computational cost of training can be reduced through the following strategies: Network pruning: Pruning neurons or convolutional kernels that contribute less can reduce the number of parameters and computational volume of the network, achieving the purpose of reducing the computational cost; Knowledge Distillation: By training a smaller student network to mimic the behavior of a larger teacher network. This method can not only reduce the computational overhead of the model but also make the model more lightweight while retaining high performance.

[0069] Furthermore, in the embodiments of the present application, the temporal processing module performs temporal smoothing processing on the above-mentioned target video images of each frame to achieve the temporal consistency of the video images.

[0070] Since a video is composed of consecutive frames, not only each individual frame needs to be processed, but also the temporal consistency needs to be maintained. Here, temporal smoothing techniques can be adopted to reduce unnatural jumps or flickers between video frames. Among them, the temporal smoothing technique adopts the following two points: Optical flow calculation and motion compensation: By analyzing the motion between adjacent frames, motion compensation is performed to reduce the temporal inconsistency caused by interpolation.

[0071] Inter-frame interpolation: Using a neural network-based frame interpolation method to generate additional intermediate frames, making the video transition more natural when smoothed.

[0072] Furthermore, in the embodiments of the present application, the image denoising module performs denoising and detail enhancement on the video images of each frame output by the temporal processing module to obtain the enhanced target video images of each frame. Specifically, through denoising techniques, the noise and artifacts generated during the super-resolution process can be effectively reduced. The specific denoising scheme can adopt the following two points: First, using a convolutional neural network (CNN) for denoising can remove noise while enhancing the resolution.

[0073] Second, adopting a multi-scale denoising method to process the image at different scales to obtain a clearer output.

[0074] Finally, in the embodiments of the present application, through the image synthesis module, the enhanced target video images of each frame are synthesized to generate a target video file with a second resolution, that is, a 4K resolution video is output. Here, by synthesizing each enhanced image and re-encoding it into a target video file in 4K video format, common video formats are H.264, H.265 and other formats. Further, quality evaluation and optimization can be performed on the output result of the video image enhancement module. The specific implementation steps of this stage can be executed in the following manner: During the process of super-resolution improvement, indicators such as peak signal-to-noise ratio and structural similarity index can be used to automatically detect the improved effect, and the detected parameters are passed to the VDSR model for continuous learning.

[0075] The display of the embodiments of the present application builds a video image enhancement module by introducing a preprocessing module, an image enhancement module, a timing processing module, an image denoising module and an image synthesis module. Through the collaborative work of the above modules, low-resolution video images are converted into high-resolution video images, effectively improving the video resolution; at the same time, through image enhancement, timing processing and denoising processing, the signal-to-noise ratio and overall quality of the video are improved, ensuring the coherence between frames of the reconstructed video, making the video easier to watch and analyze, and being beneficial to further improving the display image processing performance.

[0076] Next, the image processing method provided by the present application will be described. The image processing method described below can be correspondingly referred to the display for image processing described above.

[0077] Figure 4 is a schematic flowchart of the image processing method provided by the embodiments of the present application, which can be applied to any of the aforementioned displays, such as Figure 4 As shown, the method includes: Step S1, receiving an externally input image magnification instruction; the image magnification instruction carries the area position information of the area to be processed of the image to be processed displayed on the display panel; Step S2, identifying the content type of the target image block corresponding to the area position information; Step S3, in response to the image magnification instruction, magnifying the target image block according to the magnification method corresponding to the content type, and displaying the processed image information.

[0078] It can be understood that the detailed implementation manners of the above method can refer to the introduction in the foregoing embodiments of each unit / module, and will not be elaborated here.

[0079] It should be understood that the method in the above embodiments is applied to the above display. The implementation principle and technical effects of this method are similar to the description of the functions of the corresponding program modules of the above display. The implementation process of this method can specifically refer to the working process of the above display, which will not be elaborated here.

[0080] The image processing method of the embodiments of the present application optimizes the internal hardware structure of the display by introducing an instruction input interface, an image recognition module, and an image magnification processing module, enabling the display to have the function of image processing locally. Thus, there is no need to consider the compatibility issues between traditional image processing software and system platforms. At the same time, by identifying the content type of the images displayed on the display and specifically calling the matching magnification method for image magnification processing, it can effectively eliminate the dependence on external image processing software and platforms. Meanwhile, it can conveniently achieve efficient processing of the images displayed on the display and enhance the user experience.

[0081] Based on the content of the above embodiments, as an alternative embodiment, the content type includes text type and non-text type; step S3, in response to an image magnification instruction, magnify the target image block according to the magnification method corresponding to the content type, including: In response to an image magnification instruction, when it is determined that the content type is text type, a local pixel processing and compensation mechanism is used to magnify the text information included in the target image block; Or, In response to an image magnification instruction, when it is determined that the content type is non-text type, the target image block is magnified according to the magnification method corresponding to the image magnification multiple.

[0082] Based on the content of the above embodiments, as an alternative embodiment, in response to an image magnification instruction, when it is determined that the content type is non-text type, the target image block is magnified according to the magnification method corresponding to the image magnification multiple, including: In response to an image magnification instruction, when it is determined that the content type is non-text type and the image magnification multiple is less than the target magnification threshold, the first image magnification method is used to magnify the target image block; Or, In response to an image magnification instruction, when it is determined that the content type is non-text type and the image magnification multiple is not less than the target magnification threshold, the second image magnification method is used to magnify the target image block.

[0083] Figure 5 is a schematic structural diagram of the image processing system provided by the embodiments of the present application. As Figure 5 shown, the system includes the display 100, the instruction input device 200, and the image source 300 described in any one of the foregoing; The instruction input device 200 is respectively connected to the instruction input interface 101 of the image source 300 and the display 100; the display panel 104 of the display 100 is connected to the image source 300; The instruction input device 200 is used to control the image source 300 to transmit the image to be processed to the display panel 104 of the display for display; The instruction input device 200 is further used for the user to input an image magnification instruction for the image to be processed, and transmit the image magnification instruction to the instruction input interface 101 of the display 100; the image magnification instruction carries the area position information of the area to be processed of the image to be processed; The image recognition module 102 of the display 100 is used to recognize the content type of the target image block corresponding to the area position information; The image magnification processing module 103 of the display 100 is used to respond to the image magnification instruction, magnify the target image block according to the magnification method corresponding to the content type, and send the processed image information to the display panel 104 for display.

[0084] Specifically, the image source in the embodiment of the present application may be a server, or a device with an operating system such as a personal computer (PC); the instruction input device may include input devices such as a mouse and a keyboard.

[0085] In the embodiment of the present application, on the one hand, the instruction input device is connected to the image source, and the display panel of the display is connected to the image source. In this way, the user can operate the image source through the instruction input device to obtain the image to be processed, and transmit the image to the display panel of the display for display.

[0086] On the other hand, the instruction input device is connected to the instruction input interface of the display. In this way, for the image to be processed displayed on the display, the user can input an image magnification instruction for the image to be processed through the instruction input device, and transmit the image magnification instruction to the instruction input interface of the display. Thus, the display can enter the image magnification mode under the control of the image magnification instruction. In the image magnification mode, the user can perform targeted input operations on the image to be processed displayed on the display panel of the display.

[0087] At the same time, through the input operation of the front-end user, the image magnification instruction can carry the area position information of the area to be processed of the image to be processed. Further, in the embodiments of the present application, the image recognition module inside the display can identify the target image block corresponding to the area position information and the content type corresponding to the image information according to the area position information required to be processed in the image carried by the image magnification instruction. Furthermore, in response to the image magnification instruction, the image magnification processing module magnifies the target image block according to the magnification method corresponding to the content type, and sends the processed image information to the display panel for display.

[0088] The system of the embodiments of the present application constructs a display image processing system through an instruction input device, an image source and a display, enabling various images provided by the image source to be efficiently processed locally on the display without considering the compatibility issues between traditional image processing software and system platforms, and effectively eliminating the dependence on external image processing software and platforms. At the same time, by identifying the content type of the displayed image by the display and specifically calling the matching magnification method for image magnification processing, the effect and efficiency of display image processing can be further improved, and the user experience can be enhanced.

[0089] Based on the method in the above embodiments, the embodiments of the present application provide a computer program product, which, when running on a processor, causes the processor to execute the method in the above embodiments.

[0090] It can be understood that the processor in the embodiments of the present application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. The general-purpose processor can be a microprocessor or any conventional processor.

[0091] The method steps in the embodiments of the present application can be implemented in a hardware manner or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in a random access memory (RAM), flash memory, read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), register, hard disk, removable hard disk, CD-ROM, or any other form of storage medium well-known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0092] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, image source, or data center to another website, computer, image source, or data center in a wired manner (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or a wireless manner (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that the computer can access or a data storage device such as an image source or data center that includes one or more integrated available media. The available medium can be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)), etc.

[0093] It can be understood that the various numerical numbers involved in the embodiments of the present application are only for convenience of description and are not used to limit the scope of the embodiments of the present application.

[0094] It should be understood that expressions such as "including" and "may include" that can be used in this application indicate the existence of the disclosed functions, operations, or components, and do not limit one or more additional functions, operations, and components. In this application, terms such as "including" and / or "having" can be interpreted as indicating a specific characteristic, number, operation, component, component, or a combination thereof, but cannot be interpreted as excluding the existence or possibility of addition of one or more other characteristics, numbers, operations, components, components, or a combination thereof.

[0095] In the description of the embodiments of this application, it should be noted that unless otherwise clearly specified and limited, the term "connection" should be understood in a broad sense. For example, "connection" can be a detachable connection or a non-detachable connection; it can be a direct connection or an indirect connection through an intermediate medium. Among them, "fixed connection" means that the two are connected and the relative positional relationship after connection remains unchanged. "Rotational connection" means that the two are connected and can rotate relative to each other after connection. "Sliding connection" means that the two are connected and can slide relative to each other after connection. The orientation terms mentioned in the embodiments of this application, such as "top", "bottom", "inside", "outside", "left", "right", etc., are only references to the direction of the drawings. Therefore, the orientation terms used are for better and clearer description and understanding of the embodiments of this application, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the embodiments of this application.

[0096] As described above, the above are only specific embodiments of this application, but the protection scope of this application is not limited thereto. Any person skilled in the art within the technical scope disclosed in this application can easily think of changes or substitutions, which should all be covered within the protection scope of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

Claims

1. A display for image processing, characterized in that: include: Command input interface, image recognition module, image magnification processing module and display panel; The command input interface is connected to the image recognition module and the image enlargement processing module respectively; the image enlargement processing module is connected to the image recognition module and the display panel respectively; The command input interface is used to receive an externally input image magnification command; the image magnification command carries the area position information of the image to be processed displayed by the display panel that needs to be processed; The image recognition module is used to identify the content type of the target image block corresponding to the area position information; The image enlargement processing module is used to respond to the image enlargement instruction, enlarge the target image block according to the enlargement method corresponding to the content type, and send the processed image information to the display panel for display.

2. The display for image processing according to claim 1, characterized in that: The content type includes text type and non-text type; The image magnification processing module is used to use local pixel processing and compensation mechanism to magnify the text information contained in the target image block when determining that the content type is a text type; or, The image magnification processing module is used to perform image magnification processing on the target image block according to a magnification method corresponding to an image magnification factor when it is determined that the content type is a non-text type.

3. The display for image processing according to claim 2, characterized in that: The image magnification processing module is used to use a first image magnification method to perform image magnification processing on the target image block when it is determined that the content type is a non-text type and the image magnification factor is less than a target magnification threshold; or, The image magnification processing module is used to perform image magnification processing on the target image block by using a second image magnification method when it is determined that the content type is a non-text type and the image magnification factor is not less than the target magnification threshold.

4. The display for image processing according to claim 1, characterized in that: Also includes a video image enhancement module; The video image enhancement module is connected to the instruction input interface; The command input interface is also used to receive externally input video image enhancement commands; The video image enhancement module is used to respond to the video image enhancement instruction, perform video image enhancement processing on the to-be-processed video file of the first resolution specified by the video image enhancement instruction, and generate a target video file of the second resolution.

5. The display for image processing according to claim 4, characterized in that: The video image enhancement module includes a pre-processing module, an image enhancement module, a timing processing module, an image denoising module and an image synthesis module which are connected in sequence; The preprocessing module is used to perform video decoding on the video file to be processed to obtain each frame of video image contained in the video file to be processed, and perform frame alignment on each frame of the video image to obtain each aligned frame of video image; The image enhancement module is used to extract the residual image corresponding to each aligned frame of video image, and generate each frame of target video image of the second resolution based on each aligned frame of video image and the corresponding residual image; The timing processing module is used to perform timing smoothing processing on each frame of the target video image; The image denoising module is used to denoise and enhance the details of each frame of video image output by the timing processing module to obtain enhanced target video images of each frame; The image synthesis module is used to synthesize the enhanced target video images of each frame to generate a target video file with the second resolution.

6. The display for image processing according to any one of claims 1 to 5, characterized in that: Also includes: Image input interface; The image input interface is connected to an external image source, and is used to receive an image to be processed from the external image source, and transmit the image to be processed to the display panel for display; and is used to receive a video file to be processed from the external image source.

7. An image processing system, comprising a command input device and an image source, characterized in that: Also includes the display according to any one of claims 1 to 6; The command input device is connected to the image source and the command input interface of the display respectively; the display panel of the display is connected to the image source; The command input device is used to control the image source to transmit the image to be processed to the display panel of the display for display; The command input device is also used for a user to input an image magnification command for the image to be processed, and transmit the image magnification command to the command input interface of the display; the image magnification command carries the location information of the area to be processed of the image to be processed; The image recognition module of the display is used to identify the content type of the target image block corresponding to the area position information; The image magnification processing module of the display is used to respond to the image magnification instruction, magnify the target image block according to the magnification method corresponding to the content type, and send the processed image information to the display panel for display.

8. An image processing method applied to a display according to any one of claims 1 to 6, characterized in that: include: receiving an externally input image magnification instruction; the image magnification instruction carries the location information of the area to be processed of the image to be processed displayed by the display panel; Identifying a content type of a target image block corresponding to the region position information; In response to the image magnification instruction, the target image block is magnified according to the magnification method corresponding to the content type, and the processed image information is displayed.

9. The image processing method according to claim 8, characterized in that: The content type includes a text type and a non-text type; and in response to the image magnification instruction, performing magnification processing on the target image block according to the magnification method corresponding to the content type, including: In response to the image magnification instruction, when it is determined that the content type is a text type, amplifying the text information contained in the target image block by using a local pixel processing and compensation mechanism; or, In response to the image magnification instruction, when it is determined that the content type is a non-text type, image magnification processing is performed on the target image block according to a magnification method corresponding to the image magnification factor.

10. The image processing method according to claim 9, characterized in that: In response to the image magnification instruction, when it is determined that the content type is a non-text type, performing image magnification processing on the target image block according to a magnification method corresponding to the image magnification factor, including: In response to the image magnification instruction, when it is determined that the content type is a non-text type and the image magnification factor is less than a target magnification threshold, performing image magnification processing on the target image block in a first image magnification manner; or, In response to the image magnification instruction, when it is determined that the content type is a non-text type and the image magnification factor is not less than the target magnification threshold, a second image magnification method is used to perform image magnification processing on the target image block.

Images