Image processing method and device, video processing method and device

Abstract

The embodiment of the present specification relates to the technical field of image processing, and particularly relates to an image processing method and device, a video processing method and device, which solve the problem of large amount of calculation of a traditional image super-resolution method. The image processing method performs convolution processing based on wavelet transformation on the basis of luminance channel data of a to-be-processed image to obtain luminance channel super-resolution data of the to-be-processed image, and generates a super-resolution image corresponding to the to-be-processed image on the basis of the luminance channel super-resolution data. The wavelet transformation has the characteristic of multi-scale, and can more accurately observe the multi-scale information of a signal. Therefore, the convolution processing based on wavelet transformation can extract more features on the basis of the luminance channel data to improve the accuracy of image processing. In addition, the embodiment of the present specification can realize the super-resolution processing of the to-be-processed image by using the convolution processing based on wavelet transformation, the method is simple, the amount of calculation is small, and the computing cost is greatly reduced.

Description

TECHNICAL FIELD

[0001] The embodiments of the present specification relate to the technical field of image processing, in particular to an image processing method and an image processing device, a video processing method and a video processing device. BACKGROUND

[0002] Image super-resolution technology can process a low-resolution image into a high-resolution image, and thus is widely applied in many scenes such as live broadcast, real-time communication and film restoration. For example, the image super-resolution technology is used to improve the video quality in a live broadcast scene, to improve the image and / or video with deteriorated quality after multiple forwarding in a real-time communication scene, and to improve the classic film with rough quality in a film restoration scene. However, the traditional image super-resolution method is complex in scheme and large in device cost, and thus cannot be implemented under limited device conditions. SUMMARY

[0003] Therefore, the embodiments of the present specification provide an image processing method and an image processing device, and a computer readable storage medium and an electronic device, which solve the problem of large calculation amount of the traditional image super-resolution method.

[0004] In a first aspect, an image processing method is provided, which includes: performing wavelet transform-based convolution processing based on luminance channel data of a to-be-processed image to obtain luminance channel super-resolution data of the to-be-processed image; and generating a super-resolution image corresponding to the to-be-processed image based on the luminance channel super-resolution data.

[0005] In a second aspect, a video processing method is provided, which includes: processing a video frame of a to-be-processed video by using the image processing method mentioned in the first aspect to obtain a super-resolution image corresponding to the video frame of the to-be-processed video; and generating super-resolution video data of the to-be-processed video based on the super-resolution image corresponding to the video frame of the to-be-processed video.

[0006] In a third aspect, an image processing device is provided, which includes: a luminance super-resolution processing module configured to perform wavelet transform-based convolution processing based on luminance channel data of a to-be-processed image to obtain luminance channel super-resolution data of the to-be-processed image; and a super-resolution image generation module configured to generate a super-resolution image corresponding to the to-be-processed image based on the luminance channel super-resolution data.

[0007] In a fourth aspect, a video processing device is provided, which includes: a video processing module configured to process a video frame of a to-be-processed video by using the image processing method mentioned in the first aspect to obtain a super-resolution image corresponding to the video frame of the to-be-processed video; and a super-resolution video generation module configured to generate super-resolution video data of the to-be-processed video based on the super-resolution image corresponding to the video frame of the to-be-processed video.

[0008] In a fifth aspect, an embodiment of the present specification provides a computer-readable storage medium, the storage medium storing instructions, when the instructions are executed by a processor of an electronic device, the electronic device is enabled to perform the image processing method mentioned in the first aspect, and / or the video processing method mentioned in the second aspect.

[0009] In a sixth aspect, an embodiment of the present specification provides an electronic device, the electronic device comprising: a processor; a memory for storing computer executable instructions; the processor is configured to execute the computer executable instructions to implement the image processing method mentioned in the first aspect, and / or the video processing method mentioned in the second aspect.

[0010] In a seventh aspect, an embodiment of the present specification provides a neural network model, comprising: a luminance channel super-resolution network comprising a wavelet transform-based convolution module, wherein the wavelet transform-based convolution module is configured to perform wavelet transform-based convolution on luminance channel data of a to-be-processed image to obtain luminance channel super-resolution data of the to-be-processed image.

[0011] In an eighth aspect, an embodiment of the present specification provides a computer program product, the computer program product comprising instructions, when the instructions are executed by a processor of a computer device, the computer device is enabled to perform the image processing method mentioned in the first aspect, and / or the video processing method mentioned in the second aspect.

[0012] The image processing method provided by the embodiments of the present specification performs wavelet transform-based convolution on the luminance channel data of the to-be-processed image to obtain luminance channel super-resolution data of the to-be-processed image, and then generates a super-resolution image corresponding to the to-be-processed image based on the luminance channel super-resolution data. The wavelet transform has the characteristic of multi-scale, and can simultaneously observe the time and frequency information of the signal, so that the multi-scale information of the signal can be observed more accurately. Therefore, performing wavelet transform-based convolution on the luminance channel data can extract more features based on the luminance channel data to obtain more accurate luminance channel super-resolution data. In addition, the embodiments of the present specification can realize super-resolution processing of the to-be-processed image by using wavelet transform-based convolution, which is simple and has small calculation amount, thereby greatly reducing the calculation cost. BRIEF DESCRIPTION OF DRAWINGS

[0013] Figure 1 Fig. 1 shows an application scenario of the image processing method provided by an embodiment of the present specification.

[0014] Figure 2 Fig. 2 shows an application scenario of the image processing method provided by another embodiment of the present specification.

[0015] Figure 3Fig. 1 shows an application scenario of the image processing method according to an embodiment of the present specification.

[0016] Figure 4 Fig. 2 shows a flowchart of the image processing method according to an embodiment of the present specification.

[0017] Figure 5 Fig. 3 shows a structure diagram of the luminance channel super-resolution network according to an embodiment of the present specification.

[0018] Figure 6 Fig. 4 shows a structure diagram of the luminance channel super-resolution network according to another embodiment of the present specification.

[0019] Figure 7 Fig. 5 shows a structure diagram of the luminance channel super-resolution network according to another embodiment of the present specification.

[0020] Figure 8 Fig. 6 shows a structure diagram of the luminance channel super-resolution network according to another embodiment of the present specification.

[0021] Figure 9 Fig. 7 shows a flowchart of the image processing method according to another embodiment of the present specification.

[0022] Figure 10 Fig. 8 shows a structure diagram of the super-resolution network according to an embodiment of the present specification.

[0023] Figure 11a Fig. 9 shows a structure diagram of the chroma super-resolution network according to an embodiment of the present specification.

[0024] Figure 11b Fig. 10 shows a structure diagram of the chroma super-resolution network according to another embodiment of the present specification.

[0025] Figure 12 Fig. 11 shows a structure diagram of the super-resolution network according to another embodiment of the present specification.

[0026] Figure 13 Fig. 12 shows a flowchart of the image processing method according to another embodiment of the present specification.

[0027] Figure 14 Fig. 13 shows a structure diagram of the super-resolution network according to another embodiment of the present specification.

[0028] Figure 15 Fig. 14 shows a structure diagram of the super-resolution network according to another embodiment of the present specification.

[0029] Figure 16 Fig. 15 shows a structure diagram of the super-resolution network according to another embodiment of the present specification.

[0030] Figure 17 Fig. 1 shows a structural schematic diagram of an image processing device according to an embodiment of the present specification.

[0031] Figure 18 Fig. 2 shows a structural schematic diagram of an image processing device according to another embodiment of the present specification.

[0032] Figure 19 Fig. 3 shows a structural schematic diagram of an image processing device according to another embodiment of the present specification.

[0033] Figure 20 Fig. 4 shows a structural schematic diagram of an electronic device according to an embodiment of the present specification. DETAILED DESCRIPTION

[0034] The technical solutions in the embodiments of the present specification will be described clearly and completely below in combination with the drawings in the embodiments of the present specification. Obviously, the described embodiments are only part of the embodiments of the present specification, rather than all the embodiments. Based on the embodiments in the embodiments of the present specification, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the protection scope of the embodiments of the present specification.

[0035] The technical solutions in the embodiments of the present specification will be described clearly and completely below in combination with the drawings in the embodiments of the present specification. Obviously, the described embodiments are only part of the embodiments of the present specification, rather than all the embodiments. Based on the embodiments in the embodiments of the present specification, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the protection scope of the embodiments of the present specification. Figures 1 to 3 The application scenarios are exemplified below. In the following application scenarios, the function of super-resolution processing of images or videos is realized by performing convolution processing based on wavelet transform on the luminance channel data of the to-be-processed image. More specifically, the convolution processing based on wavelet transform on the luminance channel data refers to performing inner product on the luminance channel data corresponding to each pixel point in the to-be-processed image and the convolution kernel, so as to perform feature extraction and analysis on the luminance channel data corresponding to each pixel point, i.e., similar to the multi-scale analysis in the frequency domain of the image by wavelet transform, so as to finally obtain the luminance channel super-resolution data of the to-be-processed image.

[0036] Figure 1 Fig. 1 shows a structural schematic diagram of an image processing device according to an embodiment of the present specification. Figure 1As shown, the application scenario of this embodiment involves a client at the video sending end and a client at the video receiving end. Specifically, the client at the video sending end acquires the video to be processed (i.e., executes step S110) and sends the video to be processed to the client at the video receiving end (i.e., executes step S120). Correspondingly, the client at the video receiving end receives the video to be processed and performs wavelet transform-based convolution processing on the luminance channel data of the video to be processed to obtain luminance channel super-resolution data of the video to be processed. Then, based on the luminance channel super-resolution data, it generates super-resolution images corresponding to the video frames of the video to be processed, thereby generating super-resolution video data corresponding to the video to be processed (i.e., executes step S130), and plays the super-resolution video data (i.e., executes step S140).

[0037] In some other embodiments, the processing of the video to be processed (i.e., step S130) can also be performed on the client side of the video sending end. That is, the client side of the video sending end processes the video and then sends it to the client side of the video receiving end, so that the client side of the video receiving end can directly play the super-resolution video data.

[0038] More specifically, the application scenario mentioned in this embodiment can be a live streaming scenario, where the client of the video sending end can be the broadcaster's client, and the client of the video receiving end can be the viewer's client. More specifically, the application scenario mentioned in this embodiment can be a video conferencing scenario, where the client of the video sending end can be the client of the first participant, and the client of the video receiving end can be the client of the second participant. Furthermore, it is understood that the video sending end can also be the video receiving end, and vice versa; this will not be elaborated upon in this embodiment.

[0039] To further clarify the processing procedure of the video receiver's client for the video to be processed, the following will combine... Figure 2 Further examples will be provided. For instance... Figure 2 As shown, the video receiving client includes a video receiver 210, a processor 220, and a video player 230. Specifically, the video receiver 210 receives the video to be processed and sends it to the processor 220. The processor 220 performs wavelet transform-based convolution processing on the luminance channel data of the video to be processed to obtain luminance channel super-resolution data. Then, based on the luminance channel super-resolution data, it generates super-resolution images corresponding to the video frames of the video to be processed, thereby generating super-resolution video data corresponding to the video to be processed, and sends the super-resolution video data to the video player. The video player 230 plays the super-resolution video data. The processor 220 can be hardware such as a processor on the client, and the video player 230 can be hardware such as a display on the client.

[0040] In some embodiments, the client can be a terminal such as a mobile phone, a notebook computer, and a tablet computer, and can also be a terminal such as a smart television, a virtual reality (VR) headset, and smart glasses.

[0041] In addition to the application scenarios described in the above embodiments, the technical solutions of the present specification are also applicable to communication scenarios implemented based on Real-Time Communication (RTC) technology. RTC technology refers to a communication technology capable of transmitting and receiving text, audio, and video in real time, and is applicable to scenarios such as live streaming, on-demand, video conferencing, online classroom, online chat room, and game interaction, and realizes real-time transmission of pure audio data, video data, and the like. The technical solutions of the present application can be specifically applied to communication scenarios such as live streaming, on-demand, video conferencing, online classroom, online chat room, and game interaction implemented based on RTC.

[0042] The following will be described in combination with Figure 3 Further examples are provided.

[0043] Figure 3 As shown, the application scenario of the image processing method provided by another embodiment of the present specification is shown. As shown in the figure, Figure 3 As shown, the application scenario of the image processing method provided by the embodiment of the present specification includes a server 310 and a plurality of clients 320. Figure 3 Only client A, client B, and client C are shown. The plurality of clients 320 can establish a communication connection through the server 310, and in an RTC scenario, the server 310 is used to provide RTC services between the plurality of clients 320. The plurality of clients 320 can respectively act as a sending end or a receiving end, and realize real-time communication through the server 310.

[0044] The user can interact with the server 310 through the client 320 to receive data sent by other clients 320, or send data to other clients 320, and the like. In an RTC scenario, the user can publish a data stream to the server 310 through the client 320, and the server 200 pushes the data stream to the client subscribing to the data stream. The data stream can be media data such as a video stream. For example, in a live streaming scenario, the anchor user can collect media data in real time through the client 320 and send it to the server 310. The media data of different anchor users is distinguished through a live streaming room, and the server 310 can push the media data of the anchor user to the viewing user entering the live streaming room corresponding to the anchor user. For another example, in a conference scenario, the participant user can collect media data in real time through the client 320 and send it to the server 310, and the server 310 can push the media data sent by each client 320 to the client 320 of other participant users, and the like.

[0045] The data transmitted by the client 320 can need to be encoded, transcoded, compressed, and the like before being published to the server 310, and the image can be super-resolution processed according to the technical solutions of the embodiments of the present specification, and the like, which will be described in detail below.

[0046] The client 320 and the server 310 are connected through a network. The network provides a medium for a communication link between the client 320 and the server 310. The network can include various connection types, such as wired, wireless communication links, or fiber optic cables, and the like.

[0047] The client 320 can be a browser, an application (APP), or a web application such as a Hyper Text Markup Language 5 (H5) application, or a light application (also known as a small program, a lightweight application), or a cloud application, and the like. The client 320 can be developed based on a software development kit (SDK) of a corresponding service provided by the server, such as an RTC SDK, and the like. The client 320 can be deployed in an electronic device and needs to rely on the device or some APP in the device to run, and the like. The electronic device can have a display screen and support information browsing, and the like, such as a personal mobile terminal such as a mobile phone, a tablet computer, a personal computer, and the like. Various other types of applications can also be configured in the electronic device, such as human-computer dialogue applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, and the like.

[0048] The server 310 can include servers that provide various services, such as servers that provide communication services for multiple clients, servers that provide support for models used on the client for background training, servers that process data sent by the client, and the like.

[0049] It should be noted that the server 310 can be implemented as a distributed server cluster composed of multiple servers, or as a single server. The server can also be a server of a distributed system, or a server combined with a blockchain. The server can also be a cloud server of cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, and the like. basic cloud computing services, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0050] It should be noted that the image processing method or the video processing method provided in the embodiments of the present specification is generally executed by a server, and the corresponding image processing device or video processing device is generally arranged in the server. However, in other embodiments of the present application, the client can also have similar functions as the server, so as to execute the image processing method or the video processing method provided in the embodiments of the present specification. In other embodiments, the image processing method or the video processing method provided in the embodiments of the present specification can also be executed by the client and the server together.

[0051] Figure 4 A flowchart of an image processing method provided by an embodiment of the present specification is shown. As shown in the figure, the image processing method comprises the following steps. Figure 4

[0052] Step 410, performing wavelet transform-based convolution processing based on the luminance channel data of the to-be-processed image to obtain luminance channel super-resolution data of the to-be-processed image.

[0053] The to-be-processed image can be a video frame in a video or a general image.

[0054] The to-be-processed image corresponds to YUV three-channel data, wherein the Y channel data is also called luminance channel data, which can represent luminance information and detail texture information, and the UV channel data is also called chrominance channel data, which can represent color information. Correspondingly, the luminance channel super-resolution data refers to the data obtained by performing super-resolution processing on the luminance channel data of the to-be-processed image. It can be understood that the resolution corresponding to the luminance channel super-resolution data is higher than the resolution corresponding to the luminance channel data.

[0055] Exemplarily, a luminance channel super-resolution network containing a wavelet-like transform module can be used to perform wavelet transform-based convolution processing based on the luminance channel data of the to-be-processed image to obtain the luminance channel super-resolution data of the to-be-processed image. The luminance channel super-resolution network is a neural network capable of performing super-resolution processing on the luminance channel data, and the wavelet-like transform module can be a component module of the luminance channel super-resolution network, which can include a plurality of convolution layers and deconvolution layers.

[0056] Step 420, generating a super-resolution image corresponding to the to-be-processed image based on the luminance channel super-resolution data.

[0057] Exemplarily, after obtaining the luminance channel super-resolution data of the to-be-processed image, the luminance channel super-resolution data can be fused with the chrominance channel data of the to-be-processed image to obtain a super-resolution image of the to-be-processed image. It can be understood that the chrominance channel data of the to-be-processed image can also be super-resolution processed to obtain chrominance channel super-resolution data of the to-be-processed image, and then the luminance channel super-resolution data and the chrominance channel super-resolution data of the to-be-processed image are fused to obtain a super-resolution image corresponding to the to-be-processed image.​

[0058] In the super-resolution task, the Y channel data is more concerned than the UV channel data. Therefore, the embodiments of the present specification prefer to use the luminance channel super-resolution network comprising the wavelet-like transform module to process the luminance channel data of the to-be-processed image, to obtain luminance channel super-resolution data of the to-be-processed image, and then generate a super-resolution image corresponding to the to-be-processed image based on the luminance channel super-resolution data. The wavelet transform has the characteristic of multi-scale, which can simultaneously observe the time and frequency information of the signal, so that the multi-scale information of the signal can be observed more accurately. Therefore, the convolution processing based on the wavelet transform based on the luminance channel data can extract more features based on the luminance channel data to obtain more accurate luminance channel super-resolution data. In addition, the convolution processing based on the wavelet transform can realize the super-resolution processing of the to-be-processed image, which is simple in method and small in calculation amount, greatly reducing the calculation overhead.

[0059] The specific functions of the wavelet-like transform module will be described in detail below.

[0060] In an embodiment of the present specification, the wavelet-like transform module is configured to: perform convolution operation of simulated wavelet transform based on the luminance channel data to obtain luminance feature data; perform feature analysis based on the luminance feature data to obtain luminance analysis data; and perform deconvolution operation of simulated inverse wavelet transform based on the luminance analysis data to obtain luminance channel super-resolution data. Specifically, the wavelet-like transform module can comprise a wavelet transform unit, a first luminance analysis unit and a wavelet inverse transform unit. The wavelet transform unit is configured to perform convolution operation of simulated wavelet transform based on the luminance channel data to obtain luminance feature data. The first luminance analysis unit is configured to perform feature analysis based on the luminance feature data to obtain luminance analysis data. The wavelet inverse transform unit is configured to perform deconvolution operation of simulated inverse wavelet transform based on the luminance analysis data to obtain luminance channel super-resolution data. The wavelet transform unit, the first luminance analysis unit and the wavelet inverse transform unit can each comprise one or more convolution layers and one or more deconvolution layers. The convolution parameters of the convolution layer can be represented in the form of W[output channel number x input channel number x convolution kernel size x convolution kernel size]. The convolution parameters of the deconvolution layer can be represented in the form of W[input channel number x output channel number x convolution kernel size x convolution kernel size].

[0061] In practical applications, for example, Figure 5As shown, the input of the wavelet transform module can be the luminance channel data with the dimension of 1x1x180x320 (i.e., a set of data with the depth of 1 and the resolution of 180x320). The wavelet transform unit can be a convolution layer Conv with the convolution parameters of W[16x1x2x2] (i.e., the output channel number of the convolution layer Conv is 16, the input channel number is 1, and the convolution kernel size is 2x2), and thus the wavelet transform unit can perform feature extraction on the 1-channel luminance channel data by using convolution with the convolution kernel size of 2x2 to obtain 16-channel luminance feature data. The first luminance analysis unit can be a convolution layer Conv with the convolution parameters of W[16x16x3x3] (i.e., the output channel number of the convolution layer Conv is 16, the input channel number is 16, and the convolution kernel size is 3x3), and thus the first luminance analysis unit can perform feature analysis on the 16-channel luminance feature data by using convolution with the convolution kernel size of 3x3 to obtain 16-channel luminance feature analysis data. The inverse wavelet transform unit can be a convolution layer ConvTranspose with the convolution parameters of W[16x1x2x2] (i.e., the input channel number of the convolution layer ConvTranspose is 16, the output channel number is 1, and the convolution kernel size is 2x2), and thus the inverse wavelet transform unit can perform deconvolution operation on the 16-channel luminance feature analysis data by using convolution with the convolution kernel size of 2x2 to obtain 1-channel luminance channel super-resolution data, i.e., the luminance channel super-resolution data with the dimension of 1x1x360x540 (i.e., a set of data with the depth of 1 and the resolution of 360x540) can be output.

[0062] It can be understood that, Figure 5 The convolution parameters of the convolution layer and the deconvolution layer in the embodiments shown can further include a bias B [dimension] Figure 5 For example, the first luminance analysis unit can be a convolution layer Conv with the convolution parameters of W[16x16x3x3], B

[16] (i.e., the output channel number of the convolution layer Conv is 16, the input channel number is 16, the convolution kernel size is 3x3, and the bias is 16 dimensions). Figure 5 The convolution parameters of the convolution layer and the deconvolution layer in the embodiments shown can be other numerical values, which are not specifically limited in the embodiments of the present application. Figure 5 There can be an activation function layer (not shown) between the convolution layers and between the convolution layer and the deconvolution layer in the embodiments shown. Figure 5 The activation function layer can be Relu.

[0063] The convolution parameters of the wavelet transform unit and the wavelet inverse transform unit can also include a step length, which can be set to 2. The convolution parameters of the first brightness analysis unit can also include a step length, which can be set to 1. The wavelet transform unit is set to a three-dimensional step length of 2, which can reduce the width and height of the to-be-processed image processed by the wavelet transform unit to one half of the original, which is equivalent to down-sampling, so that the convolution operation can be performed on the reduced to-be-processed image to extract the deep (i.e., high-dimensional) detail information (i.e., high-frequency information) and approximate information (i.e., low-frequency information) of the image. For example, the width and height of the to-be-processed image before being processed by the wavelet transform unit are 180x320, and the width and height of the to-be-processed image after being processed by the wavelet transform unit are 90x160. The wavelet inverse transform unit is set to a step length of 2, which can expand the width and height of the to-be-processed image processed by the wavelet inverse transform unit to twice the original, which is equivalent to up-sampling, so that the enlarged image can be displayed on a display device with higher resolution. For example, the width and height of the to-be-processed image before being processed by the wavelet inverse transform unit are 90x160, and the width and height of the to-be-processed image after being processed by the wavelet transform unit are 180x320.

[0064] In combination with the description in the above embodiments, it can be known that the wavelet-like transform module in the above embodiments has a simple structure, and the calculation amount of using the wavelet-like transform module to perform super-resolution processing on the to-be-processed image is small, thereby further reducing the calculation overhead.

[0065] The more specific functions of the wavelet-like transform module are described in detail below.

[0066] In an embodiment of the present disclosure, the feature analysis based on the brightness feature data obtains brightness analysis data, including: performing feature analysis based on the brightness feature data with a first convolution kernel size to obtain first analysis data; performing feature analysis based on the first analysis data with a second convolution kernel size to obtain second analysis data; and performing feature analysis based on the second analysis data with a third convolution kernel size to obtain the brightness analysis data.

[0067] Specifically, the first brightness analysis unit is further configured to: perform feature analysis based on the brightness feature data with a first convolution kernel size to obtain first analysis data; perform feature analysis based on the first analysis data with a second convolution kernel size to obtain second analysis data; and perform feature analysis based on the second analysis data with a third convolution kernel size to obtain the brightness analysis data.

[0068] In actual applications, for example, Figure 6As shown, the input of the wavelet transform module can be the luminance channel data with the dimension of 1x8x180x320 (i.e., a set of data with the depth of 8 and the resolution of 180x320). The wavelet transform unit can be a convolution layer Conv with the convolution parameters of W[16x8x2x2] (i.e., the output channel number of the convolution layer Conv is 16, the input channel number is 8, and the convolution kernel size is 2x2), so that the wavelet transform unit can perform feature extraction on the 8-channel luminance channel data by using convolution with the convolution kernel size of 2x2 to obtain 16-channel luminance feature data. The first luminance analysis unit can include three convolution layers, and the convolution parameters of the three convolution layers are respectively a convolution layer Conv with the convolution parameters of W[16x16x1x1], B

[16] (i.e., the output channel number of the convolution layer Conv is 16, the input channel number is 16, the convolution kernel size is 1x1, and the bias is 16-dimensional), a convolution layer Conv with the convolution parameters of W[16x16x3x3], B

[16] (i.e., the output channel number of the convolution layer Conv is 16, the input channel number is 16, the convolution kernel size is 3x3, and the bias is 16-dimensional), and a convolution layer Conv with the convolution parameters of W[16x16x1x1], B

[16] (i.e., the input channel number of the convolution layer Conv is 16, the output channel number is 16, the convolution kernel size is 1x1, and the bias is 16-dimensional). Therefore, the first luminance analysis unit can perform feature analysis on the 16-channel luminance feature data by using convolution with the convolution kernel size of 1x1 to obtain 16-channel first analysis data, then perform feature analysis on the 16-channel first analysis data by using convolution with the convolution kernel size of 3x3 to obtain 16-channel second analysis data, and finally perform feature analysis on the 16-channel second analysis data by using convolution with the convolution kernel size of 1x1 to obtain 16-channel luminance analysis data.

[0069] It can be understood that Figure 6 The convolution parameters of the convolution layers and the deconvolution layers in the embodiments shown can also be other numerical values, which are not specifically limited in the embodiments of the present application. Figure 6 In the embodiments shown, there can also be an activation function layer between the convolution layers. The activation function layer can be Relu. The embodiments of the present application do not specifically limit the type of the activation function of the activation function layer.

[0070] The first luminance analysis unit in the above embodiments includes three convolution layers, so that the image to be processed can be analyzed three times, and the accuracy of image processing is further improved to obtain more accurate luminance channel super-resolution data.

[0071] In order to further improve the accuracy of image processing, the convolution processing based on wavelet transform is performed on the brightness channel data of the image to be processed to obtain brightness channel super-resolution data of the image to be processed, including: performing the convolution processing based on wavelet transform and residual processing on the brightness channel data of the image to be processed to obtain the brightness channel super-resolution data.

[0072] Exemplarily, the residual processing mentioned above is implemented by means of a residual module, that is, the brightness channel super-resolution network can further include a residual module. Then, the convolution processing based on wavelet transform and residual processing on the brightness channel data of the image to be processed to obtain the brightness channel super-resolution data can include: processing the brightness channel data by using the brightness channel super-resolution network including the wavelet-like transform module and the residual module to obtain the brightness channel super-resolution data. Specifically, the residual module is used for convolution operation of feature extraction and feature analysis based on the brightness channel data. The residual module can include: a first convolution unit, an activation function unit, a second convolution unit and a summation unit.

[0073] In actual application, as shown in Figure 7 , the brightness channel super-resolution network includes a wavelet-like transform module 711 and a residual module 712, in addition, the brightness channel super-resolution network further includes an input layer 713, a convolution layer 714, a summation layer 715, a convolution layer 716, a summation layer 717, a convolution layer 718, an activation function layer 719, a deconvolution layer 720 and an output layer 721. The wavelet-like transform module 711 can be the structure of the embodiment shown in Figure 5 or Figure 6 , and also can be the structure shown in Figure 7The structure of the illustrated embodiment. The residual module 712 includes a first convolution unit, an activation function unit, a second convolution unit, and a summation unit. The first convolution unit can be a convolution layer Conv with convolution parameters W [8x8x3x3], B [8] (i.e., the output channel number of the convolution layer Conv is 8, the input channel number is 8, the convolution kernel size is 3x3, and the bias is 8-dimensional). The activation function unit can be an activation function layer Relu, and the second convolution unit can be a convolution layer Conv with convolution parameters W [8x8x3x3], B [8] (i.e., the output channel number of the convolution layer Conv is 8, the input channel number is 8, the convolution kernel size is 3x3, and the bias is 8-dimensional). The summation unit can be a summation layer Add, which is used to perform summation processing on the output data of the wavelet-like transform module and the output data of the second convolution unit. The input layer 713 can input luminance channel data with a dimension of 1x1x180x320. The convolution parameters of the convolution layer 714 can be W [8x1x5x5], B [8]. The summation layer 715 is used to perform summation processing on the output data of the convolution layer 714 and the output data of the wavelet-like transform module 711. The convolution parameters of the convolution layer 716 can be W [8x8x3x3], B [8]. The summation layer 717 is used to perform summation processing on the output data of the convolution layer 714 and the output data of the convolution layer 716. The convolution parameters of the convolution layer 718 can be W [16x8x3x3], B

[16] . The activation function layer 719 is used to perform activation processing on the output data of the convolution layer 718. The convolution parameters of the deconvolution layer 720 can be W [16x1x2x2], B [1]. The output layer 721 can output luminance channel super-resolution data with a dimension of 1x1x360x640.

[0074] The number of wavelet-like transform modules and the number of residual modules can each be one or more. The embodiments of the present specification do not make specific limitations on the number of wavelet-like transform modules and the number of residual modules. Figure 7 Only an embodiment including one wavelet-like transform module and one residual module is shown in the above description. The embodiments of the present specification do not make specific limitations on the number of wavelet-like transform modules and the number of residual modules. In addition, the embodiments of the present specification do not make specific limitations on the connection order of the wavelet-like transform module and the residual module.

[0075] The wavelet transform-based module can perform high-dimensional feature extraction and analysis based on luminance channel data, while the residual module can perform low-dimensional feature extraction and analysis based on luminance channel data. In other words, wavelet transform-based convolution processing can perform high-dimensional feature extraction and analysis based on luminance channel data, while residual processing can perform low-dimensional feature extraction and analysis based on luminance channel data. Therefore, performing wavelet transform-based convolution processing and residual processing on the luminance channel data of the image to be processed yields luminance channel super-resolution data. This allows the obtained luminance channel super-resolution data to simultaneously reference both high-dimensional and low-dimensional features of the luminance channel data, thereby further improving the image quality corresponding to the obtained luminance channel super-resolution data.

[0076] To better adapt to input images of different sizes, in one embodiment of this specification, the luminance channel super-resolution network further includes a depth-adjusted convolution module. Specifically, before performing wavelet transform-based convolution processing on the luminance channel data of the image to be processed to obtain the luminance channel super-resolution data of the image to be processed, the network further includes: performing depth adjustment processing on the initial channel data of the image to be processed to obtain the luminance channel data. The aforementioned depth adjustment processing on the initial channel data of the image to be processed can be implemented using the depth-adjusted convolution module. The output layer depth of the depth-adjusted convolution module is determined based on the resolution information of the image to be processed.

[0077] Specifically, the initial channel data can be single-channel data with a shallow depth from the image to be processed, such as luminance channel data with a shallow depth. After processing by the depth adjustment convolution module, the shallow luminance channel data can be converted into luminance channel data with a greater depth. The output layer depth of the depth adjustment convolution module can be determined based on the resolution information of the image to be processed. If the resolution of the image to be processed is high, the output layer depth determined by the depth adjustment convolution module can be small. If the resolution of the image to be processed is low, the output layer depth determined by the depth adjustment convolution module can be large. This specification does not specifically limit the value of the output layer depth determined by the depth adjustment convolution module in its embodiments.

[0078] In practical applications, such as Figure 7 As shown, the depth-adjusted convolution module can be a convolutional layer 714. The convolution parameters of the convolutional layer 714 can be W[8x1x5x5]. That is, the depth of the input layer (the depth of the initial channel data) of the convolutional layer 714 is 1, and the depth of the output layer (the depth of the luminance channel data) is 8.

[0079] The output layer depth of the depth adjustment convolution module is determined based on resolution information of the to-be-processed image. In the case that the resolution of the input to-be-processed image is large, the output layer depth determined by the depth adjustment convolution module is small, thereby reducing the complexity of the brightness channel super-resolution network and reducing the problem of slow image processing speed caused by the large resolution of the input to-be-processed image.

[0080] Figure 7 The parameters of each module in the brightness channel super-resolution network of the embodiment can also be other values, and the embodiment of the present specification is not specifically limited. The following describes another parameter setting mode of each module in the brightness channel super-resolution network based on the embodiment of the present specification. Figure 7 Figure 8 The parameters of each module in the brightness channel super-resolution network of the embodiment can also be other values, and the embodiment of the present specification is not specifically limited. The following describes another parameter setting mode of each module in the brightness channel super-resolution network based on the embodiment of the present specification. Figure 8 As shown in FIG. 7, the input layer 713 can input the brightness channel data with the dimension of 1x1x540x960. The convolution parameters of the convolution layer 714 can be W[4x1x5x5], B[4]. The wavelet-like transformation module 711 includes: the convolution layer Conv:W(8x4x2x2), the convolution layer Conv:W(8x8x1x1), B[8], the activation layer Relu, the convolution layer Conv:W(8x8x3x3), B[8], the activation layer Relu, the convolution layer Conv:W(8x8x1x1), B[8], the activation layer Relu, and the deconvolution layer ConvTranspose:W(8x4x2x2) connected in series. The sum layer 715 is used for sum processing of the output data of the convolution layer 714 and the output data of the wavelet-like transformation module 711. The residual module 712 includes: the convolution layer Conv:W(4x4x3x3), B[4], the activation layer Relu, the convolution layer Conv:W(4x4x3x3), B[4], and the sum layer Add connected in series. The convolution parameters of the convolution layer 716 can be W[4x4x3x3], B[4]. The sum layer 717 is used for sum processing of the output data of the convolution layer 714 and the output data of the convolution layer 716. The convolution parameters of the convolution layer 718 can be W[8x4x3x3], B[8]. The activation function layer 719 is used for activation processing of the output data of the convolution layer 718. The convolution parameters of the deconvolution layer 720 can be W[8x1x2x2], B[1]. The output layer 721 can output the brightness channel super-resolution data with the dimension of 1x1x1080x1920.

[0081] The processing method of the brightness channel data of the to-be-processed image is described in detail above. The following describes the processing method of the chroma channel data of the to-be-processed image in combination with Figure 9 and Figure 10 The processing method of the brightness channel data of the to-be-processed image is described in detail above. The following describes the processing method of the chroma channel data of the to-be-processed image in combination with

[0082] ​In an embodiment of the present disclosure, the step of generating the super-resolution image corresponding to the to-be-processed image based on the luminance channel super-resolution data comprises the following steps.

[0083] At step 910, chroma feature extraction is performed based on the chroma channel data of the to-be-processed image to obtain chroma coarse super-resolution data.

[0084] Specifically, as shown in FIG. 10, the chroma feature extraction can be performed by the first chroma analysis module 1010 based on the chroma channel data of the to-be-processed image to obtain the chroma coarse super-resolution data. Figure 10

[0085] The chroma feature extraction performed based on the chroma channel data of the to-be-processed image to obtain the chroma coarse super-resolution data comprises: performing full-band feature extraction based on the chroma channel data to obtain full-band feature data; and performing feature analysis based on the full-band feature data to obtain the chroma coarse super-resolution data.

[0086] In actual applications, the first chroma analysis module 1010 can comprise an input unit, a full-band feature extraction unit and a full-band feature analysis unit connected in sequence. The full-band feature extraction unit is configured to perform full-band feature extraction based on the chroma channel data to obtain full-band feature data. The full-band feature analysis unit is configured to perform feature analysis based on the full-band feature data to obtain the chroma coarse super-resolution data.

[0087] The input unit Input(UV) can input the chroma channel data (i.e., UV channel data) with the dimension of 1x2x270x480. The full-band feature extraction unit can be a convolution layer Conv with the convolution parameters of W[8x2x3x3] and B[8]. The full-band feature analysis unit can be a deconvolution layer ConvTranspose with the convolution parameters of W[8x4x2x2] and B[8]. The full-band feature analysis unit can further comprise a convolution layer Conv with the convolution parameters of W[8x8x3x3] and B[8] and a deconvolution layer ConvTranspose with the convolution parameters of W[8x4x2x2] and B[8]. The convolution layer and the deconvolution layer can be followed by an activation function layer Relu. The full-band feature extraction unit can further comprise a larger number of convolution layers, and the number of the convolution layers comprised by the full-band feature extraction unit is not specifically limited in the embodiments of the present disclosure. The full-band feature analysis unit can further comprise a larger number of convolution layers and deconvolution layers, and the number of the convolution layers and deconvolution layers comprised by the full-band feature analysis unit is not specifically limited in the embodiments of the present disclosure.

[0088] At step 920, super-resolution guide data of the chroma channel is generated according to the output data of the convolution processing based on the wavelet transform.

[0089] Specifically, the super-resolution guide data is used to represent the detailed texture information of the to-be-processed image.​

[0090] In practical applications, such as Figure 10 As shown, the super-resolution guiding data for the chroma channel can be generated by the guiding data generation module 1020 based on the output data of the wavelet transform-like module. The guiding data generation module 1020 can be a convolutional layer Conv with convolutional parameters W[4x16x1x1] and B[4]. The guiding data generation module 1020 can also be a concatenated convolutional layer Conv with convolutional parameters W[4x16x1x1] and B[4] and an activation function layer ReLU.

[0091] It is understood that the output data of the wavelet transform module 711 can be further processed sequentially by the summing layer 715, the residual module 712, the convolutional layer 716, the summing layer 717, the convolutional layer 718, and the activation function layer 719 before being output to the guiding data generation module 1020. As long as the guiding data generation module 1020 generates super-resolution guiding data for the chroma channel based on the output data of the wavelet transform module 711, the implementation method in this specification does not limit whether there are other modules between the wavelet transform module 711 and the guiding data generation module 1020.

[0092] Step 930: Based on the super-resolution guided data, perform chromaticity feature analysis on the coarse chromaticity super-resolution data to obtain chromaticity channel super-resolution data.

[0093] Specifically, such as Figure 10 As shown, both the super-resolution guidance data and the coarse chroma super-resolution data can be input into the second chroma analysis module 1030, thereby enabling the second chroma analysis module 1030 to perform chroma feature extraction based on the chroma channel data of the image to be processed, and obtain chroma channel super-resolution data. The first chroma analysis module 1010, the guidance data generation module 1020, and the second chroma analysis module 1030 can form a chroma super-resolution network.

[0094] In practical applications, such as Figure 10As shown, the second chroma analysis module 1030 can include a connected sum layer Add, a convolution group, and an output layer. The sum layer Add is used to sum the output data of the first chroma analysis module 1010 and the output data of the guide data generation module 1020. The convolution group can include 2 layers of convolution, the first layer of convolution Conv has convolution parameters W[4x4x3x3], B[4], and the second layer of convolution layer Conv can be 2 convolutions with convolution parameters W[lx4x3x3], B[4], so that the output layer can output 2 chroma channel data of 1x1x540x960, i.e., U channel data and V channel data. Figure 10 The way of merging and outputting is not shown.

[0095] At step 940, based on the chroma channel super-resolution data and the luminance channel super-resolution data, a super-resolution image corresponding to the to-be-processed image is generated.

[0096] Exemplarily, the chroma channel super-resolution data and the luminance channel super-resolution data are fused together to generate a super-resolution image corresponding to the to-be-processed image.

[0097] Based on the super-resolution guide data, a chroma feature analysis operation is performed on the chroma coarse super-resolution data to obtain chroma channel super-resolution data, which can include: based on the super-resolution guide data and the chroma coarse super-resolution data, merging is performed to obtain feature merging data; based on the feature merging data, feature analysis is performed to obtain the chroma channel super-resolution data. In actual application, as shown, Figure 11a The merging function layer Concat can be used to merge the super-resolution guide data and the chroma coarse super-resolution data to obtain the feature merging data. Then the convolution layer Conv: W[n out x n in x s x s] is used to perform feature analysis on the feature merging data to obtain the chroma channel super-resolution data. n out represents the output channel number of the convolution layer Conv, n in represents the input channel number of the convolution layer Conv, and s x s represents the convolution kernel size of the convolution layer Conv.

[0098] In addition, based on the super-resolution guide data, a chroma feature analysis operation is performed on the chroma coarse super-resolution data to obtain chroma channel super-resolution data, which can include: based on the super-resolution guide data, first fine feature analysis is performed to obtain guide analysis data; based on the chroma coarse super-resolution data, second fine feature analysis is performed to obtain chroma analysis data; the guide analysis data and the chroma analysis data are summed to obtain the chroma channel super-resolution data. In actual application, as shown, Figure 11bAs shown, the first convolutional layer Conv1: W[n] can be utilized. out1 xn in1 Based on the super-resolution guided data, the first fine feature analysis is performed to obtain the guided analysis data, and the second convolutional layer Conv2 is used: W[n out2 xn in2 [x s2 x s2] Based on the coarse super-resolution chromaticity data, a second fine feature analysis is performed to obtain chromaticity analysis data. Then, the summation layer Add is used to sum the guided analysis data and the chromaticity analysis data to obtain the chromaticity channel super-resolution data. n out1 n represents the number of output channels of the convolutional layer Conv1. in1 s1 represents the number of input channels of convolutional layer Conv1, and s1 x s1 represents the kernel size of convolutional layer Conv1. out2 n represents the number of output channels of the Conv2 convolutional layer. in2 s2 represents the number of input channels of the convolutional layer Conv2, and s2 x s2 represents the kernel size of the convolutional layer Conv2.

[0099] Figure 11a and Figure 11b Both implementations can achieve the same super-resolution effect through parameter settings. For example, the convolutional layer Conv can be split into convolutional layer Conv1 and convolutional layer Conv2. For instance, n can be set... out =n out1 +n out2 n in =n in1 =n in2 s = s1 = s2. If the convolution parameters of the convolutional layer Conv can also include a bias B [dimension], the bias B [dimension] of the convolutional layer Conv can be split into two, namely B1 [dimension] and B2 [dimension]. B1 [dimension] can be the bias of the convolutional layer Conv1, and B2 [dimension] can be the bias of the convolutional layer Conv2. The specific values ​​of the convolution parameters of the convolutional layers Conv, Conv1, and Conv2 can be selected according to the actual situation, and this specification does not impose specific limitations on the implementation method. In addition, the convolutional layers Conv, Conv1, and Conv2 can also be followed by an activation function layer ReLU. Other activation function layers can also be followed by the convolutional layers Conv, Conv1, and Conv2, and this specification does not impose specific limitations on the implementation method.

[0100] Since the chroma channel data lacks detailed texture information, the first chroma analysis module 1010 and the second chroma analysis module 1030 only process the input chroma channel data, which may not accurately infer the details or edge contour positions of the image, resulting in color cast of the inferred chroma channel super-resolution data or color overflow due to misalignment of the luminance channel data and the chroma channel data. Therefore, the embodiments of the present specification generate super-resolution guide data for the chroma channel based on the output data of the convolution processing based on wavelet transform, and then perform chroma feature analysis operations on the chroma coarse super-resolution data based on the super-resolution guide data to obtain the chroma channel super-resolution data, so that the inference on the chroma channel data refers to the detailed texture information in the luminance channel data, thereby reducing color cast and color overflow.

[0101] The luminance channel data and the chroma channel data of the to-be-processed image can be merged together and input into the luminance channel super-resolution network and the chroma channel super-resolution network, and the luminance channel super-resolution network is used to process the luminance channel data of the to-be-processed image, and the chroma channel super-resolution network (which can include the first chroma analysis module 1010 and the second chroma analysis module 1030) is used to process the chroma channel data of the to-be-processed image, and therefore, the luminance channel data and the chroma channel data merged together need to be processed to generate the luminance channel data and the chroma channel data.

[0102] In an embodiment of the present specification, as shown in Figure 12 If the luminance channel data and the chroma channel data of the to-be-processed image are merged together and input into the luminance channel super-resolution network and the chroma channel super-resolution network, the luminance channel data can be split out by using the splitting module 1210, processed by using the luminance channel super-resolution network, and then the chroma channel data can be split out by using the downsampling module 1220, processed by using the chroma channel super-resolution network. The splitting module 1210 can include a convolution layer with a convolution parameter of W[lx3x3x3]. The downsampling module 1220 can include a convolution layer with a convolution parameter of W[2x3x2x2]. The step parameter of the convolution layer with a convolution parameter of W[2x3x2x2] included in the downsampling module 1220 can be set to 2, so as to realize one-half downsampling.

[0103] The graphics processing unit (GPU) or neural-network processing unit (NPU) has strong parallel processing capability. Therefore, the image processing method of the embodiments of the present specification is executed by using the GPU or NPU. In the case of merging the luminance channel data and the chrominance channel data together to input the luminance channel super-resolution network and the chrominance channel super-resolution network, the parallel processing capability of the GPU or NPU can be well exerted, the luminance channel super-resolution network and the chrominance channel super-resolution network can process the luminance channel data and the chrominance channel data in parallel, so as to reduce the processing time of the luminance channel data and the chrominance channel data and improve the image processing efficiency.

[0104] Based on the strong parallel processing capability of the GPU or NPU, the following will be combined with the description of the embodiments of the present specification to further improve the image processing efficiency. Figure 13 The embodiments for further improving the image processing efficiency will be described in detail.

[0105] In an embodiment of the present specification, as shown in FIG. 13, before the convolution processing based on the wavelet transform is performed on the luminance channel data of the image to be processed to obtain the luminance channel super-resolution data of the image to be processed, the following steps are further included. Figure 13

[0106] Step 1310: performing image region-based division on the initial luminance channel data of the image to be processed to obtain N luminance data units.

[0107] Specifically, N is a positive integer greater than 1. The initial luminance channel data of the image to be processed can be 1x1x540x960 (i.e., 1 group of data with a depth of 1 and a resolution of 540x960). The image region-based division on the initial luminance channel data of the image to be processed can be dividing the initial luminance channel data into multiple blocks based on the image region of the image to be processed. For example, taking the initial luminance channel data 1x1x540x960 divided into 2 blocks (i.e., N equals to 2) as an example, the luminance data units obtained by the division can be 2 pieces of 1x1x540x480 data (i.e., 960 is divided into 2 parts, each part is 480).

[0108] ​In order to prevent the subsequent processing of N luminance data units respectively, and the image obtained by assembling again appears obvious boundary line, the adjacent luminance data units can be overlapped when dividing the initial luminance channel data. For example, the initial luminance channel data corresponds to a to-be-processed image with a dimension of 1xcxhxw. The to-be-processed image is divided into two parts according to the width (w) dimension, and two luminance data units are obtained, which are 1xcxhx[0:(w / 2+lace)] and 1xcxhx[(w / 2-lace):w] respectively. c, h, and w represent the depth, height, and width of the to-be-processed image respectively. lace represents the radius of the overlapping area in the width dimension. 0:(w / 2+lace) represents the size between the start and end of the width dimension, the start is 0, and the end is (w / 2+lace). (w / 2-lace):w represents the size between the start and end of the width dimension, the start is (w / 2-lace), and the end is w. The area overlapping length between the two luminance data units is 2*lace. lace is greater than or equal to the radius of the overall receptive field of the network, which can ensure that the boundary line is avoided.

[0109] Exemplarily, taking the to-be-processed image 1x1x540x960 corresponding to the initial luminance channel data as an example, c=1, h=540, w=960, and lace=16. The two luminance data units are 1x1x540x496 and 1x1x540x496 respectively.

[0110] In step 1320, the N luminance data units are merged in the depth dimension to obtain merged luminance data.

[0111] Exemplarily, taking the two luminance data units 1x1x540x496 and 1x1x540x496 obtained above as an example, after merging in the depth dimension, the merged luminance data is 1x2x540x496.

[0112] In step 1330, data extraction is performed on the merged luminance data to obtain the luminance channel data of each of the N luminance data units.

[0113] Specifically, as Figure 14As shown (taking N = 2 as an example), the merged luminance data can be subjected to data extraction by the extraction module 1410 to obtain luminance channel data of each of the N luminance data units. The extraction module 1410 can include a first extraction unit 1411 and a second extraction unit 1412. The first extraction unit 1411 is configured to extract a first luminance data unit from the merged luminance data, and the second extraction unit 1412 is configured to extract a second luminance data unit from the merged luminance data. The first extraction unit 1411 can be a convolution layer with a convolution parameter of W[lx2xlxl]. The second extraction unit 1412 can also be a convolution layer with a convolution parameter of W[lx2xlxl].

[0114] It can be understood that the extraction module 1410 includes an extraction unit equal to N in number. That is, the number of luminance data units can be the same as the number of extraction units set. The extraction units included in the extraction module 1410 can all be convolution layers with a convolution parameter of W[lx2xlxl].

[0115] Based on the luminance channel data of the image to be processed, the convolution processing based on wavelet transform is performed to obtain the luminance channel super-resolution data of the image to be processed, including:

[0116] In step 1340, the convolution processing based on wavelet transform is respectively performed based on the luminance channel data of each of the N luminance data units to obtain the luminance channel super-resolution data of the image to be processed.

[0117] Specifically, the convolution processing based on wavelet transform is respectively performed based on the luminance channel data of each of the N luminance data units to obtain the luminance channel super-resolution data of the image to be processed, which can be that the luminance channel super-resolution network is used to process the luminance channel data of each of the N luminance data units, thereby obtaining the luminance channel super-resolution data of each of the N luminance data units, and then the luminance channel super-resolution data of each of the N luminance data units is spliced to obtain the luminance channel super-resolution data of the image to be processed.

[0118] By performing image region-based division on the initial luminance channel data of the image to be processed, N luminance data units are obtained, and then the N luminance data units are merged in the depth dimension to obtain merged luminance data, so that the merged luminance data can be input into the luminance channel super-resolution network. Then, the luminance channel super-resolution network performs data extraction on the merged luminance data to obtain luminance channel data of each of the N luminance data units, and finally, the convolution processing based on wavelet transform is respectively performed based on the luminance channel data of each of the N luminance data units to obtain the luminance channel super-resolution data of the image to be processed. Thus, the parallel processing capability of the GPU or NPU can be fully utilized to improve the efficiency of image processing.

[0119] To further utilize the parallel processing capability of the GPU or NPU and improve the efficiency of image processing, in an embodiment of the present specification, the convolution processing based on wavelet transform is performed on the luminance channel data of each of the N luminance data units respectively to obtain the luminance channel super-resolution data of the to-be-processed image, which can include: performing the convolution processing based on wavelet transform on the luminance channel data of each of the N luminance data units in parallel to obtain the luminance channel super-resolution data of each of the N luminance data units, and then generating the luminance channel super-resolution data of the to-be-processed image based on the luminance channel super-resolution data of each of the N luminance data units.

[0120] In actual application, as shown in FIG. 14B, the luminance channel data of two luminance data units output by the first extraction unit 1411 and the second extraction unit 1412 can be processed in parallel. Figure 14

[0121] The above describes the embodiment of processing the luminance channel data of N luminance data units in parallel, and the following describes the embodiment of processing the luminance channel data of N luminance data units in series. Figure 15 In an embodiment of the present specification, the convolution processing based on wavelet transform is performed on the luminance channel data of each of the N luminance data units respectively to obtain the luminance channel super-resolution data of the to-be-processed image, which includes: performing the convolution processing based on wavelet transform on the luminance channel data of each of the N luminance data units in series to obtain the luminance channel super-resolution data of each of the N luminance data units, and then generating the luminance channel super-resolution data of the to-be-processed image based on the luminance channel super-resolution data of each of the N luminance data units.

[0122] In actual application, as shown in FIG. 14B, the luminance channel data of two luminance data units output by the first extraction unit 1411 and the second extraction unit 1412 can be processed in parallel. Figure 15

[0123] As shown in FIG. 14B, the luminance channel data of the luminance data unit extracted by the first extraction unit 1411 can be processed first, and then the luminance channel data of the luminance data unit extracted by the second extraction unit 1412 can be processed, and so on until the luminance channel data of the luminance data unit extracted by all the extraction units are processed, so as to realize the processing in series. Figure 14 Figure 15 It can be understood that the parallel processing mode shown in FIG. 14A and the series processing mode shown in FIG. 14B can be used in the same embodiment, for example, the extraction module 1410 includes four extraction units, and the luminance channel data of the luminance data units extracted by the four extraction units are represented by S1, S2, S3 and S4 respectively. S1 and S2 can be processed in parallel to obtain the processing result {S1, S2}, S3 and S4 can be processed in parallel to obtain the processing result {S3, S4}, and then {S1, S2} and {S3, S4} can be processed in series.

[0124] In the above description, the luminance channel data of the luminance data units extracted by the extraction units are processed in parallel or in series, but the present specification is not limited thereto. Figure 15 ​​​Based on the serial processing mode shown, in order to further improve the accuracy of image processing, a serial guidance module 1413 can be provided. In an embodiment of the present specification, the wavelet transform-based convolution processing is performed on the respective luminance channel data of the N luminance data units in series to obtain the luminance channel super-resolution data of the respective luminance data units, including: for each luminance data unit that is not processed for the first time in the N luminance data units, determining the super-resolution guidance data of the luminance data unit based on the luminance channel super-resolution data of the luminance data unit that is processed immediately before the luminance data unit; and performing wavelet transform-based convolution processing on the super-resolution guidance data of the luminance data unit and the luminance channel data of the luminance data unit to obtain the luminance channel super-resolution data of the luminance data unit.

[0125] As shown, for the second luminance data unit, the serial guidance module 1413 can be used to determine the super-resolution guidance data of the second luminance data unit based on the luminance channel super-resolution data of the first luminance data unit. Then, the luminance channel super-resolution network is used to process the super-resolution guidance data of the second luminance data unit and the luminance channel data of the second luminance data unit to obtain the luminance channel super-resolution data of the second luminance data unit. Figure 15

[0126] By providing the serial guidance module 1413, when processing each luminance data unit that is not processed for the first time, the luminance channel super-resolution data of the luminance data unit that is processed immediately before the luminance data unit is referred to, thereby further improving the accuracy of image processing.

[0127] The parallel processing mode shown in Figure 14 and the serial processing mode shown in Figure 15 will be described in detail below. Another embodiment will be described in detail in combination with Figure 16 In an embodiment of the present specification, the wavelet transform-based convolution processing is performed on the respective luminance channel data of the N luminance data units to obtain the luminance channel super-resolution data of the image to be processed, including: splicing the respective luminance channel data of the N luminance data units in the depth dimension to obtain luminance channel spliced data with a depth dimension of N*P; performing wavelet transform-based convolution processing on the luminance channel spliced data to obtain luminance channel super-resolution data with a depth dimension of N*P; and splicing the luminance channel super-resolution data with the depth dimension of N*P to obtain the luminance channel super-resolution data of the image to be processed.

[0128] Specifically, the luminance channel data of each luminance data unit has a dimension of P, and P is a positive integer. The wavelet transform-based convolution processing on the luminance channel spliced data to obtain the luminance channel super-resolution data with the depth dimension of N*P can be performed by using the luminance channel super-resolution network to process the luminance channel spliced data to obtain the luminance channel super-resolution data with the depth dimension of N*P.​

[0129] Taking N=2 and P=1 as an example, the dimension of the luminance channel data of each luminance data unit can be 1x1x540x496. Concatenating the luminance channel data of two luminance data units along the depth dimension yields a luminance channel concatenated data with a dimension of 1x2x540x496. Using a luminance channel super-resolution network to process the concatenated luminance channel data, the resulting luminance channel super-resolution data can have a dimension of 1x2x1080x992. Finally, assembling the luminance channel super-resolution data yields a luminance channel super-resolution data of 1x1x1080x1920 for the image to be processed.

[0130] The implementation method of this manual ( Figure 16 The dimension of the luminance channel super-resolution data of the image to be processed obtained after the assembly (as shown in the implementation) can be adjusted by setting the convolution parameters of each convolutional and deconvolutional layer, achieving the same result as without partitioning (e.g., Figure 8 and Figure 10 The dimensionality of the super-resolution data of the brightness channel of the image to be processed obtained in the embodiment shown is the same.

[0131] The following section explains how to implement concatenated convolution for single-layer convolution, and then explains how to assemble the luminance channel super-resolution data with a depth dimension of N*P to obtain the luminance channel super-resolution data of the image to be processed.

[0132] Exemplarily, the dimension of the input IN of the convolution layer is 1xn1xhxw, and the dimension of the output OUT after the convolution layer is 1xn2xh1xw1 (for example, the step of the convolution layer is 1, h=h1, w=w1), and the convolution kernel of the convolution layer is kernel[weight, bias]. Wherein, weight and bias are both parameters of the convolution kernel. The dimension of weight is n2xn1xkxk, k is the width and height of weight, and the dimension of bias is n2. Then, find a convolution kernel kernel1[weight1, bias1], the dimension of weight1 is (2xn2)x(2xn1)xkxk, and the dimension of the convolution kernel bias1 is 2xn2. weight1 and bias1 can be filled in weight and bias (that is, the convolution kernel with weight1 is assembled by the convolution kernel with weight). For example: fill weight1[0:n2,0:n1,:,:] and weight1[n2:2xn2,n1:2xn1,:,:] into weight[:,:,:,:], and fill bias1[0:n2] and bias1[n2:2xn2] into bias[:]. After filling, it can be known that weight1 is not filled at this time, and weight1[0:n2,n1:2xn1,:,:] and weight1[n2:2xn2,0:n1,:,:] are still empty. If all these empty parts are filled with 0, kernel1[weight1, bias1] satisfies: the dimension of two inputs IN1 and IN2 is 1xn1xhxw, the combined input INC with the dimension of 1x(2xn1)xhxw is obtained by stacking IN1 and IN2 in the second dimension (depth dimension), and the dimension of the output OUTC after the convolution calculation of kernel1 is 1x(2xn2)xh1xw1. Denote the outputs of IN1 and IN2 after the convolution calculation of kernel as OUT1 and OUT2 respectively, OUTC[:,0:n1,:,:]=OUT1, OUTC[:,n1:2xn1,:,:]=OUT2. The above symbol “:” represents a value that can be determined according to actual needs.

[0133] It can be understood that the method of filling weight into weight1 is not unique. For example, it can also be filled as follows: weight1[0:n2, num1, :,:] = weight[:, i1, :,:], weight1[n2:2xn2, num2, :,:] = weight[:, i2, :,:], where 0 <= i1, i2 <= n1-1, and num1 and num2 are set to satisfy 0 <= num1, num2 <= 2xn1-1 and num1 and num2 are not equal to each other. Finally, as long as OUTC[:, 0:n1, :,:] = OUT1 and OUTC[:, n1:2xn1, :,:] = OUT2 are satisfied.

[0134] It can be understood that the way of filling weight into weight1 can be determined according to actual needs, as long as the following conditions are met: "{OUTC[:, Num1, :,:]} (a total of n2 Num1) constitutes OUT1 (OUTC[:, Num1, :,:] = OUT1[:, j1, :,:], j1 = 0, 1, 2,..., n2-1), {OUTC[:, Num2, :,:]} (a total of n2 Num2) constitutes OUT2 (OUTC[:, Num2, :,:] = OUT1[:, j2, :,:], j2 = 0, 1, 2,..., n2-1), Num1 and Num2 satisfy: 0 <= Num1, Num2 <= 2xn2-1 and Num1 and Num2 are not equal to each other", and the embodiments of the present specification are not limited. The way of filling bias into bias1 becomes bias1[Num1] = bias[j1], bias[Num2] = bias[j2]. As long as it conforms to the above-mentioned "convolution splicing" idea, the embodiments of the present specification do not limit the specific way of splicing.

[0135] It can be understood that the splicing can not be limited to two convolutions, but can be more than two convolutions.

[0136] For the above-mentioned filling of weight into weight1, weight1 is not filled, and the above-mentioned unfilled part is directly filled with 0, which is equivalent to the part filled with 0 not playing any role, which is relatively wasteful. In order to also utilize this part and improve the algorithm effect of spliced convolution, the unfilled part can not be filled with 0, which will be described in detail below.

[0137] The calculation of the unfilled part can be understood as calculating the correlation between IN1 and IN2 input to the split convolution. If there is complementary information between IN1 and IN2, the complementary information can be analyzed for correlation, and the correlation analysis result can be passed to the final output. For example, IN1 and IN2 are two adjacent frames of a video, and have a high degree of similarity. Then, there is a pixel area in IN1 that has unclear or incomplete information due to noise or blur. The pixel area at the corresponding position in IN2 can not be seriously noisy or blurred, or even not. Then, the information of the area in IN2 can be extracted to supplement the corresponding area in IN1, so as to improve the quality of IN1. Similarly, IN1 can also supplement the relevant information to IN2 to improve the quality of IN2. Through training of the network composed of convolution layers, the network composed of convolution layers can identify the correlation of the input data (for example, IN1 and IN2, or other input data). In actual application, the greater the correlation of the input data, the better the output quality. That is, the quality of OUTC[:,0:n1,:,:] in the above embodiment exceeds that of OUT1, and the quality of OUTC[:,n1:2xn1,:,:] exceeds that of OUT2.

[0138] It can be understood that training the network composed of convolution layers can be only training the part of weight1 that is not filled with weight. In order to improve the training effect, the training samples can include samples with high information correlation, samples with medium information correlation, samples with low information correlation, and samples with zero information correlation.

[0139] The above detailed implementation of using split convolution to realize multi-frame image input and multi-frame image output. Similarly, the above embodiment can also be applied to the case of multi-frame cutting. The corresponding position cut is split into a convolution split network, and each group of cuts is processed by a split network. Finally, the outputs of each split network are split into the final multi-frame output. The above embodiment significantly reduces the video processing time and improves the output video quality.

[0140] The embodiment of the present specification also provides a video processing method, comprising: processing the video frames of the video to be processed by using the above-mentioned image processing method to obtain the super-resolution image corresponding to the video frames of the video to be processed; and generating super-resolution video data of the video to be processed based on the super-resolution image corresponding to the video frames of the video to be processed. After obtaining the super-resolution video data, the super-resolution video data can be played out, so as to realize real-time processing and real-time playing.

[0141] This specification also provides a neural network model, including a luminance channel super-resolution network containing a wavelet-like transform module. The wavelet-like transform module is used to perform wavelet-like transform-based convolution processing on the luminance channel data of the image to be processed, obtaining luminance channel super-resolution data of the image to be processed. The specific structure of the wavelet-like transform module can be referred to the above embodiments and will not be repeated here. The specific structure of the luminance channel super-resolution network can be referred to the above embodiments and will not be repeated here.

[0142] The above text combined Figures 1 to 16 This specification describes in detail the method embodiments of the implementation methods. The following is in conjunction with... Figures 17 to 19 This specification describes in detail the apparatus embodiments of the implementation methods. It should be understood that the descriptions of the method embodiments correspond to the descriptions of the apparatus embodiments; therefore, any parts not described in detail can be found in the foregoing method embodiments.

[0143] Figure 17 The diagram shown is a structural schematic of an image processing apparatus provided in one embodiment of this specification. Figure 17 As shown, the image processing apparatus 1700 of the embodiments of this specification includes a luminance super-resolution processing module 1710 and a super-resolution image generation module 1720.

[0144] Specifically, the luminance super-resolution processing module 1710 is configured to perform wavelet transform-based convolution processing on the luminance channel data of the image to be processed to obtain luminance channel super-resolution data of the image to be processed. The super-resolution image generation module 1720 is configured to generate a super-resolution image corresponding to the image to be processed based on the luminance channel super-resolution data. In one embodiment of this specification, performing wavelet transform-based convolution processing on the luminance channel data of the image to be processed to obtain luminance channel super-resolution data includes: performing a simulated wavelet transform convolution operation on the luminance channel data to obtain luminance feature data; performing feature analysis on the luminance feature data to obtain luminance analysis data; and performing a simulated inverse wavelet transform deconvolution operation on the luminance analysis data to obtain luminance channel super-resolution data. In one embodiment of this specification, performing feature analysis on the luminance feature data to obtain luminance analysis data includes: performing feature analysis on the luminance feature data with a first convolution kernel size to obtain first analysis data; performing feature analysis on the first analysis data with a second convolution kernel size to obtain second analysis data; and performing feature analysis on the second analysis data with a third convolution kernel size to obtain luminance analysis data.

[0145] The luminance super-resolution processing module 1710 is further configured to perform wavelet transform-based convolution processing and residual processing on the luminance channel data of the image to be processed to obtain luminance channel super-resolution data. For example, the luminance channel super-resolution network includes a residual module in addition to the wavelet transform-like module.

[0146] The luminance super-resolution processing module 1710 is further configured to obtain the luminance channel data by performing a depth adjustment processing on the initial channel data of the to-be-processed image. Illustratively, the luminance channel super-resolution network further comprises a depth adjustment convolution module. The output layer depth of the depth adjustment convolution module is determined based on the resolution information of the to-be-processed image.

[0147] Figure 18 Fig. 2 shows a structural schematic diagram of an image processing apparatus provided by another embodiment of the present specification. As shown in Fig. 2, the super-resolution image generation module 1720 of the embodiment of the present specification comprises a chroma extraction unit 1721, a luminance guidance unit 1722, a chroma super-resolution unit 1723, and a fusion unit 1724. Figure 18

[0148] Specifically, the chroma extraction unit 1721 is configured to perform a chroma feature extraction operation based on the chroma channel data of the to-be-processed image to obtain chroma coarse super-resolution data. The luminance guidance unit 1722 is configured to generate super-resolution guidance data of the chroma channel according to the output data of the convolution processing based on the wavelet transform, wherein the super-resolution guidance data is used to represent the detailed texture information of the to-be-processed image. The chroma super-resolution unit 1723 is configured to perform a chroma feature analysis operation on the chroma coarse super-resolution data based on the super-resolution guidance data to obtain chroma channel super-resolution data. The fusion unit 1724 is configured to generate a super-resolution image corresponding to the to-be-processed image based on the chroma channel super-resolution data and the luminance channel super-resolution data.

[0149] The chroma extraction unit 1721 is further configured to perform full-band feature extraction based on the chroma channel data to obtain full-band feature data, and perform feature analysis based on the full-band feature data to obtain the chroma coarse super-resolution data.

[0150] The chroma super-resolution unit 1723 is further configured to merge the super-resolution guidance data and the chroma coarse super-resolution data to obtain feature merged data, and perform feature analysis based on the feature merged data to obtain the chroma channel super-resolution data. Alternatively, the chroma super-resolution unit 1723 is further configured to perform first fine feature analysis based on the super-resolution guidance data to obtain guidance analysis data, perform second fine feature analysis based on the chroma coarse super-resolution data to obtain chroma analysis data, and sum the guidance analysis data and the chroma analysis data to obtain the chroma channel super-resolution data.

[0151] Figure 19 Fig. 3 shows a structural schematic diagram of an image processing apparatus provided by another embodiment of the present specification. As shown in Fig. 3, the image processing apparatus 1700 of the embodiment of the present specification further comprises a division module 1730, a merging module 1740, and an acquisition module 1750. Figure 19

[0152] ​​Specifically, the dividing module 1730 is configured to perform image region-based division on initial luminance channel data of the to-be-processed image to obtain N luminance data units, where N is a positive integer greater than 1. The merging module 1740 is configured to merge the N luminance data units in the depth dimension to obtain merged luminance data. The obtaining module 1750 is configured to perform data extraction on the merged luminance data to obtain luminance channel data of each of the N luminance data units.

[0153] The luminance super-resolution processing module 1710 is further configured to perform wavelet transform-based convolution processing on the luminance channel data of each of the N luminance data units respectively to obtain luminance channel super-resolution data of the to-be-processed image.

[0154] The luminance super-resolution processing module 1710 is further configured to perform wavelet transform-based convolution processing on the luminance channel data of each of the N luminance data units in parallel respectively to obtain luminance channel super-resolution data of each of the N luminance data units; and generate luminance channel super-resolution data of the to-be-processed image based on the luminance channel super-resolution data of each of the N luminance data units.

[0155] The luminance super-resolution processing module 1710 is further configured to perform wavelet transform-based convolution processing on the luminance channel data of each of the N luminance data units in series respectively to obtain luminance channel super-resolution data of each of the N luminance data units; and generate luminance channel super-resolution data of the to-be-processed image based on the luminance channel super-resolution data of each of the N luminance data units.

[0156] The luminance super-resolution processing module 1710 is further configured to, for each luminance data unit that is not processed for the first time in the N luminance data units, determine super-resolution guide data of the luminance data unit based on luminance channel super-resolution data of a previously processed luminance data unit corresponding to the luminance data unit; and perform wavelet transform-based convolution processing on the super-resolution guide data of the luminance data unit and the luminance channel data of the luminance data unit to obtain luminance channel super-resolution data of the luminance data unit.

[0157] The luminance super-resolution processing module 1710 is further configured to splice the luminance channel data of each of the N luminance data units in the depth dimension to obtain luminance channel spliced data with a depth dimension of N*P, where the luminance channel data of each of the N luminance data units has a dimension of P, and P is a positive integer; perform wavelet transform-based convolution processing on the luminance channel spliced data to obtain luminance channel super-resolution data with a depth dimension of N*P; and splice the luminance channel super-resolution data with the depth dimension of N*P to obtain luminance channel super-resolution data of the to-be-processed image.

[0158] Figures 17 to 19The operations and functions of the brightness super-resolution processing module 1710, the super-resolution image generation module 1720, the division module 1730, the merging module 1740, and the acquisition module 1750 in the image processing apparatus provided, and the color difference extraction unit 1721, the brightness guiding unit 1722, the color difference super-resolution unit 1723, and the fusion unit 1724 included in the super-resolution image generation module 1720 can refer to the above Figures 3 to 16 The image processing method provided is not repeated here in order to avoid repetition.

[0159] The video processing apparatus provided by the embodiments of the present specification further comprises: a video processing module configured to process a video frame of a to-be-processed video by using the image processing method of the above embodiments to obtain a super-resolution image corresponding to the video frame of the to-be-processed video; and a super-resolution video generation module configured to generate super-resolution video data of the to-be-processed video based on the super-resolution image corresponding to the video frame of the to-be-processed video.

[0160] Figure 20 As shown, the electronic device 2000 comprises one or more processors 2001 and memories 2002, and computer program instructions stored in the memories 2002, which, when executed by the processors 2001, cause the processors 2001 to perform the image processing method of any of the above embodiments. Figure 20 As shown, the electronic device 2000 comprises one or more processors 2001 and memories 2002, and computer program instructions stored in the memories 2002, which, when executed by the processors 2001, cause the processors 2001 to perform the image processing method of any of the above embodiments.

[0161] The processor 2001 can be a central processing unit (CPU) or other forms of processing units having data processing and / or instruction execution capabilities, and can control other components in the electronic device to perform desired functions.

[0162] The memory 2002 can include one or more computer program products, which can include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may, for example, include random access memory (RAM), cache, and / or the like. Non-volatile memory may, for example, include read-only memory (ROM), hard disk, flash memory, and / or the like. One or more computer program instructions can be stored on the computer-readable storage medium, and the processor 2001 can execute the program instructions to implement the steps in the image processing method of the above embodiments of the present specification and / or other desired functions.

[0163] In one example, the electronic device 2000 can further include an input device 2003 and an output device 2004, which are interconnected through a bus system and / or other forms of connection (not shown in the figure). Figure 20

[0164] In addition, the input device 2003 can include, for example, a keyboard, a mouse, a microphone, etc.

[0165] The output device 2004 can output various information to the outside. The output device 2004 can include, for example, a display, a speaker, a printer, and a communication network and a remote output device connected thereto, etc.

[0166] Of course, in order to simplify, Figure 20 In the figure, only some of the components of the electronic device 2000 related to the embodiments of the present specification are shown, and components such as buses, input / output interfaces, etc. are omitted. In addition to this, the electronic device 2000 can further include any other appropriate components according to the specific application.

[0167] In addition to the above-described method and device, the embodiments of the present specification can also be a computer program product including computer program instructions that, when executed by a processor, cause the processor to perform the steps in the image processing method of any of the above-described embodiments.

[0168] The computer program product can be written in any combination of one or more programming languages, including an object-oriented programming language such as Java, C++, etc., and conventional procedural programming languages, such as the "C" programming language, or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, as a stand-alone software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0169] In addition, the embodiments of the present specification can also be a computer readable storage medium having stored thereon computer program instructions that, when executed by a processor, cause the processor to perform the steps in the image processing method according to various embodiments of the present specification described in the above "Exemplary Methods" section of the present specification.

[0170] ​The computer readable storage medium can be any combination of one or more computer readable medium(s). The computer readable medium can be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium include an electrical connection having one or more wires, a portable disk, a hard disk, a RAM, a ROM, an erasable programmable read only memory (EPROM or Flash memory), an optical fiber, a portable compact disk read only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0171] The above generally describes the basic principles of the embodiments of the present specification in conjunction with specific examples, but it should be noted that the advantages, benefits, effects and the like mentioned in the embodiments of the present specification are merely examples and are not limiting, and these advantages, benefits, effects and the like cannot be considered as necessary for each embodiment of the embodiments of the present specification. In addition, the above specific details disclosed are only for the purpose of example and understanding, and are not limiting, and the above details do not limit the embodiments of the present specification to necessarily adopt the above specific details for implementation.

[0172] The block diagrams of the devices, apparatuses, equipment, systems involved in the embodiments of the present specification are only illustrative examples and are not intended to require or imply the connection, arrangement, configuration as shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, equipment, systems can be connected, arranged, configured in any manner. Words such as "include", "contain", "have" and the like are open-ended words that mean "including but not limited to", and can be used interchangeably. The words "or" and "and" used herein mean the word "and / or", and can be used interchangeably unless the context clearly indicates otherwise. The word "such as" used herein means the phrase "such as but not limited to", and can be used interchangeably.

[0173] It should also be noted that in the apparatuses, equipment and methods of the embodiments of the present specification, each component or each step can be decomposed and / or recombined. These decompositions and / or recombinations should be considered as equivalent solutions of the embodiments of the present specification.

[0174] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application embodiments. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other aspects without departing from the scope of the present application embodiments. Thus, the present application embodiments are not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0175] The above description has been presented for the purpose of illustration and description. Furthermore, this description is not intended to limit the embodiments of the present application to the forms disclosed herein. Although various example aspects and embodiments have been discussed above, those of ordinary skill in the art will appreciate a variety of modifications, alternatives, permutations, and equivalents thereof.

[0176] The above description is intended to be illustrative and not restrictive. Many other potential applications of the present application embodiments will be apparent to those of ordinary skill in the art from the description and illustrations provided herein. It is therefore intended that all potential applications of the present application embodiments be included within the scope of the present application embodiments.

Claims

1. An image processing method, characterized in that, include: The brightness channel data of the image to be processed is subjected to wavelet transform-based convolution processing to obtain the brightness channel super-resolution data of the image to be processed. Based on the super-resolution data of the brightness channel, a super-resolution image corresponding to the image to be processed is generated; Before performing wavelet transform-based convolution processing on the luminance channel data of the image to be processed to obtain the luminance channel super-resolution data of the image to be processed, the method further includes: The initial luminance channel data of the image to be processed is divided based on the image region to obtain N luminance data units, where N is a positive integer greater than 1; The N brightness data units are merged in the depth dimension to obtain merged brightness data; Data extraction is performed on the merged brightness data to obtain the brightness channel data of each of the N brightness data units; The process of performing wavelet transform-based convolution processing on the luminance channel data of the image to be processed to obtain luminance channel super-resolution data of the image to be processed includes: The wavelet transform-based convolution process is performed on the brightness channel data of each of the N brightness data units to obtain the brightness channel super-resolution data of the image to be processed.

2. The image processing method according to claim 1, characterized in that, The process of performing wavelet transform-based convolution processing on the luminance channel data of the image to be processed to obtain luminance channel super-resolution data of the image to be processed further includes: Based on the brightness channel data, a convolution operation with simulated wavelet transform is performed to obtain brightness feature data; Based on the brightness feature data, feature analysis is performed to obtain brightness analysis data; Based on the brightness analysis data, a deconvolution operation of simulated wavelet inverse transform is performed to obtain the brightness channel super-resolution data.

3. The image processing method according to claim 2, characterized in that, The step of performing feature analysis based on the brightness feature data to obtain brightness analysis data includes: Based on the brightness feature data, feature analysis of the first convolution kernel size is performed to obtain the first analysis data; Based on the first analysis data, feature analysis of the second convolution kernel size is performed to obtain the second analysis data; Based on the second analysis data, feature analysis of the third convolution kernel size is performed to obtain the brightness analysis data.

4. The image processing method according to any one of claims 1 to 3, characterized in that, The process of performing wavelet transform-based convolution processing on the luminance channel data of the image to be processed to obtain luminance channel super-resolution data of the image to be processed further includes: Based on the luminance channel data of the image to be processed, the wavelet transform-based convolution processing and residual processing are performed to obtain the luminance channel super-resolution data.

5. The image processing method according to any one of claims 1 to 3, characterized in that, The step of generating a super-resolution image corresponding to the image to be processed based on the super-resolution data of the luminance channel includes: Chromaticity feature extraction is performed based on the chroma channel data of the image to be processed to obtain coarse chroma super-resolution data. Based on the output data of the wavelet transform-based convolution processing, super-resolution guiding data for the chroma channel is generated, wherein the super-resolution guiding data is used to characterize the detailed texture information of the image to be processed; Based on the super-resolution guided data, chromaticity feature analysis is performed on the coarse chromaticity super-resolution data to obtain chromaticity channel super-resolution data. Based on the super-resolution data of the chroma channel and the super-resolution data of the luminance channel, a super-resolution image corresponding to the image to be processed is generated.

6. The image processing method according to claim 5, characterized in that, The step of performing chroma feature extraction based on the chroma channel data of the image to be processed to obtain coarse chroma super-resolution data includes: Full-band feature extraction is performed based on the chroma channel data to obtain full-band feature data; Feature analysis is performed based on the full-band feature data to obtain the coarse chromaticity super-resolution data.

7. The image processing method according to claim 5, characterized in that, The step of performing chromaticity feature analysis on the coarse chromaticity super-resolution data based on the super-resolution guided data to obtain chromaticity channel super-resolution data includes: The super-resolution guided data and the coarse chrominance super-resolution data are merged to obtain feature merged data; Feature analysis is performed based on the feature merging data to obtain the color channel super-resolution data; Alternatively, the step of performing chroma feature analysis on the coarse chroma super-resolution data based on the super-resolution guided data to obtain chroma channel super-resolution data includes: Based on the super-resolution guidance data, a first fine feature analysis is performed to obtain guidance analysis data; A second fine feature analysis is performed based on the coarse chromaticity super-resolution data to obtain chromaticity analysis data. The super-resolution data of the chromaticity channel is obtained by summing the guided analysis data and the chromaticity analysis data.

8. The image processing method according to claim 1, characterized in that, The step of performing wavelet transform-based convolution processing on the luminance channel data of each of the N luminance data units to obtain the luminance channel super-resolution data of the image to be processed includes: The wavelet transform-based convolution process is performed in parallel on the luminance channel data of each of the N luminance data units to obtain the luminance channel super-resolution data of each of the N luminance data units. Based on the luminance channel super-resolution data of each of the N luminance data units, the luminance channel super-resolution data of the image to be processed is generated.

9. The image processing method according to claim 1, characterized in that, The step of performing wavelet transform-based convolution processing on the luminance channel data of each of the N luminance data units to obtain the luminance channel super-resolution data of the image to be processed includes: The wavelet transform-based convolution process is performed sequentially on the luminance channel data of each of the N luminance data units to obtain the luminance channel super-resolution data of each of the N luminance data units. Based on the luminance channel super-resolution data of each of the N luminance data units, the luminance channel super-resolution data of the image to be processed is generated.

10. The image processing method according to claim 9, characterized in that, The step of sequentially performing wavelet transform-based convolution processing on the luminance channel data of each of the N luminance data units to obtain the luminance channel super-resolution data of each of the N luminance data units includes: For each of the N luminance data units that is not being processed for the first time, Based on the super-resolution data of the luminance channel of the previous processed luminance data unit corresponding to the luminance data unit, the super-resolution guidance data of the luminance data unit is determined; Based on the super-resolution guiding data of the luminance data unit and the luminance channel data of the luminance data unit, the wavelet transform-based convolution processing is performed to obtain the luminance channel super-resolution data of the luminance data unit.

11. A video processing method, characterized in that, include: Using the image processing method according to any one of claims 1 to 10, video frames of a video to be processed are processed to obtain super-resolution images corresponding to the video frames of the video to be processed. Based on the super-resolution images corresponding to the video frames of the video to be processed, super-resolution video data of the video to be processed is generated.

12. An image processing apparatus, characterized in that, include: The luminance super-resolution processing module is configured to perform wavelet transform-based convolution processing on the luminance channel data of the image to be processed to obtain the luminance channel super-resolution data of the image to be processed. The super-resolution image generation module is configured to generate a super-resolution image corresponding to the image to be processed based on the super-resolution data of the brightness channel; The partitioning module is configured to partition the initial luminance channel data of the image to be processed based on image regions before performing wavelet transform-based convolution processing on the luminance channel data of the image to be processed to obtain the luminance channel super-resolution data of the image to be processed, thereby obtaining N luminance data units, where N is a positive integer greater than 1. The N brightness data units are merged in the depth dimension to obtain merged brightness data; Data extraction is performed on the merged brightness data to obtain the brightness channel data of each of the N brightness data units; The brightness super-resolution processing module is further configured to perform wavelet transform-based convolution processing on the brightness channel data of each of the N brightness data units to obtain the brightness channel super-resolution data of the image to be processed.

13. A video processing apparatus, characterized in that, include: The video processing module is configured to process video frames of a video to be processed using the image processing method according to any one of claims 1 to 10, and obtain a super-resolution image corresponding to the video frames of the video to be processed. The super-resolution video generation module is configured to generate super-resolution video data of the video to be processed based on the super-resolution images corresponding to the video frames of the video to be processed.

Images