Image processing method and device

Abstract

The embodiment of the invention provides an image processing method. The image processing method comprises the following steps: decomposing a to-be-super-divided target image into a first Y channel graph and a first UV channel graph; generating a plurality of first frequency diagrams of the first Y channel diagram by adopting a trained lightweight model on the basis of the first Y channel diagram, wherein the first frequency diagrams comprise high-frequency frequency diagrams and low-frequency frequency diagrams; obtaining a high-definition first Y channel graph corresponding to the first Y channel graph by adopting wavelet inverse transformation based on the first frequency graph; and obtaining a high-definition image of the target image based on the high-definition first Y channel graph and the first UV channel graph. According to the technical scheme provided by the embodiment of the invention, real-time super-division can be realized on the mobile terminal, the bandwidth cost of an enterprise is reduced, and the definition of the super-divided image in human eyes is effectively improved.

Description

Technical Field

[0001] Embodiments of the present application relate to the field of image processing technology, and in particular, to an image processing method, apparatus, computer device, computer-readable storage medium, and computer program product. Background Art

[0002] Super resolution is an image processing technology whose goal is to reconstruct a high-resolution image from a low-resolution image through an algorithm, so as to obtain a clearer and more detailed image.

[0003] However, current super-resolution methods will have problems such as untimely response, difficulty in extracting features, and unclear edges of the image after super-resolution when implemented on mobile devices, and the super-resolution effect is poor.

[0004] It should be noted that the above content is not necessarily prior art and is not used to limit the patent protection scope of the present application. Summary of the Invention

[0005] Embodiments of the present application provide an image processing method, apparatus, computer device, computer-readable storage medium, and computer program product to solve or alleviate one or more of the above technical problems.

[0006] One aspect of the embodiments of the present application provides an image processing method, the method comprising: Decompose a target image to be super-resolved into a first Y-channel image and a first UV-channel image; Generate a plurality of first frequency maps of the first Y-channel image based on the first Y-channel image by using a trained lightweight model, the first frequency maps including high-frequency frequency maps and low-frequency frequency maps; Obtain a high-definition first Y-channel image corresponding to the first Y-channel image by using inverse wavelet transform based on the first frequency maps; Obtain a high-definition image of the target image based on the high-definition first Y-channel image and the first UV-channel image.

[0007] Optionally, the first frequency maps include low-frequency frequency maps, horizontal high-frequency frequency maps, vertical high-frequency frequency maps, and diagonal high-frequency frequency maps.

[0008] Optionally, the method further comprises: Degrade a high-definition image dataset to obtain a low-definition image dataset; Decompose each high-definition image in the high-definition image dataset into a second Y-channel image and a second UV-channel image; Decompose the second Y-channel image into a plurality of second frequency maps by using wavelet transform; Decompose each low-definition image in the low-definition image dataset into a third Y-channel image and a third UV-channel image; Generate a number of third frequency maps of the third Y-channel map using a lightweight model based on the third Y-channel map; Train the lightweight model based on the second frequency map and the corresponding third frequency map of the second frequency map to obtain the trained lightweight model.

[0009] Optionally, the training of the lightweight model based on the second frequency map and the corresponding third frequency map of the second frequency map includes: Determine the peak signal-to-noise ratio of the second frequency map and the corresponding third frequency map as the first loss function; Determine the structural similarity index of the second frequency map and the corresponding third frequency map as the second loss function; Perform weighted summation based on the first loss function and the second loss function as the total loss function of the lightweight model, and train the lightweight model based on the total loss function.

[0010] Optionally, the method further includes: Convert the trained lightweight model into a first model file and a second model file, where the first model file is used to run on an Android mobile terminal, and the second model file is used to run on an IOS mobile terminal.

[0011] Optionally, the lightweight model includes several layers of convolution and residual connections.

[0012] Another aspect of the embodiments of the present application provides an image processing apparatus, the apparatus includes: A decomposition module for decomposing a target image to be super-resolved into a first Y-channel map and a first UV-channel map; A generation module for generating a number of first frequency maps of the first Y-channel map using the trained lightweight model based on the first Y-channel map, where the first frequency maps include high-frequency frequency maps and low-frequency frequency maps; A transformation module for obtaining a high-definition first Y-channel map corresponding to the first Y-channel map using inverse wavelet transform based on the first frequency map; A synthesis module for obtaining a high-definition image of the target image based on the high-definition first Y-channel map and the first UV-channel map.

[0013] Another aspect of the embodiments of the present application provides a computer device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein: the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method as described above.

[0014] Another aspect of the embodiments of the present application provides a computer-readable storage medium, in which computer instructions are stored, and when the computer instructions are executed by a processor, the above-described method is implemented.

[0015] Another aspect of the embodiments of the present application provides a computer program product, including a computer program, and when the computer program is executed by a processor, the above-described method is implemented.

[0016] The embodiments of the present application adopting the above technical solutions may include the following advantages: By decomposing the target image to be super-resolved into a first Y-channel map and a first UV-channel map, generating a plurality of first frequency maps of the first Y-channel map based on the first Y-channel map by using a trained lightweight model, obtaining a high-definition first Y-channel map corresponding to the first Y-channel map by using inverse wavelet transform based on the first frequency maps, and obtaining a high-definition image of the target image based on the high-definition first Y-channel map and the first UV-channel map, super-resolution processing of the image can be performed on a mobile terminal, reducing the bandwidth cost of the enterprise; by using a lightweight model to perform super-resolution and only processing the Y-channel map to generate frequency maps, the occupation of memory and computing resources during the super-resolution processing can be effectively reduced, enabling the mobile terminal to achieve real-time super-resolution; at the same time, by using wavelet transform during the process, the super-resolution processing process can pay more attention to the information in the high-frequency aspect, thereby improving the clarity of the contour edges in the image and effectively improving the clarity of the super-resolved image to the human eye. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings exemplarily show the embodiments and form a part of the description, and are used together with the written description of the description to explain the exemplary embodiments of the embodiments. The shown embodiments are only for the purpose of illustration and do not limit the scope of the claims. In all the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0018] Figure 1 Schematically shows a flowchart of an image processing method according to Embodiment 1 of the present application; Figure 2 Schematically shows a principle architecture diagram of an image processing method according to Embodiment 1 of the present application; Figure 3 Schematically shows an additional process of an image processing method according to Embodiment 1 of the present application; Figure 4 Schematically shows an example of decomposing a high-definition image into high-frequency and low-frequency images in an image processing method according to Embodiment 1 of the present application; Figure 5 Schematically shows Figure 3 a sub-step flowchart of step S210; Figure 6 Schematically shows a block diagram of an image processing apparatus according to Embodiment 2 of the present application; and Figure 7 Schematically shows a schematic diagram of the hardware architecture of a computer device according to Embodiment 3 of the present application. Detailed implementation manners

[0019] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative efforts shall fall within the protection scope of the present application.

[0020] It should be noted that the descriptions involving "first", "second", etc. in the embodiments of the present application are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one of such features. In addition, the technical solutions between the various embodiments may be combined with each other, but it must be based on the fact that those of ordinary skill in the art can implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the protection scope required by the present application.

[0021] In the description of the present application, it should be understood that the numerical labels before the steps do not identify the order of execution of the steps, but are only used to facilitate the description of the present application and distinguish each step, and thus cannot be understood as a limitation to the present application.

[0022] First, provide the term explanations involved in the present application: Super resolution: It is an image processing technology whose goal is to reconstruct a high-resolution image from a low-resolution image through an algorithm, so as to obtain a clearer and more detailed image. Simply put, it is to make a blurred picture clearer.

[0023] Y-channel map: In the field of image processing, it refers to the luminance component in the YUV color space. It represents the luminance information of the image, that is, the so-called black and white image.

[0024] UV-channel map: It is an important concept in image processing, which together with the Y-channel Figure 1 constitutes the YUV color space. If the Y-channel map represents the luminance information of the image, then the UV-channel map mainly carries the color information of the image.

[0025] Wavelet transform: It decomposes signals using a set of functions called "wavelets" as basis functions, which can capture low-frequency information over a broad time interval and high-frequency information over a narrow time interval.

[0026] Residual connection: It directly adds the input to the output of subsequent layers, allowing gradients to be more effectively backpropagated during training, thus helping to solve the vanishing gradient problem in deep networks.

[0027] Shader: A program specifically written for the GPU to execute on the Graphics Processing Unit (GPU) to efficiently calculate pixel colors, lighting effects, and other visual attributes of the rendered image.

[0028] Edge intelligence: Deploy and run AI models on device sides (such as iPhones and Android phones).

[0029] Structural Similarity Index (SSIM): It is a metric for measuring the similarity degree between two images, which is more in line with the visual perception of the human eye than the traditional Mean Squared Error (MSE). SSIM believes that the human visual system's perception of images is not only a pixel-level comparison but also pays more attention to the structural information of images, such as brightness, contrast, and structure.

[0030] Peak Signal-to-Noise Ratio (PSNR): It is a commonly used image quality evaluation metric used to measure the degree of image quality degradation during lossless compression, transmission, etc. It is obtained by comparing the Mean Squared Error (MSE) between the original image and the processed image.

[0031] Secondly, to facilitate the understanding of the technical solutions provided by the embodiments of the present application by those skilled in the art, the related technologies are described below: As the resolution of pictures and videos on the APP becomes higher and higher, the bandwidth cost of enterprises will also become higher and higher. To reduce the bandwidth cost of enterprises, super-resolution processing can be considered on terminal devices; however, the device performance of mobile terminals is generally limited, and the super-resolution algorithm consumes a large amount of memory and computing power, making it impossible for mobile terminals to respond in a timely manner if super-resolution processing is performed on the mobile side. In addition, when performing super-resolution processing, it is usually necessary to extract features from pictures or videos. However, many of the images and videos that require super-resolution on mobile terminals are small in scale and it is difficult to extract features; and currently, when performing super-resolution processing, SSIM and PSNR are usually used as evaluation criteria for the super-resolution model. However, since the human eye is more sensitive to high-frequency information in images, only SSIM and PSNR are currently used for overall evaluation, and it is impossible to make a suitable evaluation for the high-frequency information in images, often resulting in unclear edges after super-resolution processing and unsatisfactory super-resolution effects.

[0032] For this reason, the embodiments of the present application provide an image processing technical solution. In this technical solution, by performing a super-resolution model and wavelet transform on a mobile terminal through edge intelligence, high-definition pictures or videos can be generated, reducing the bandwidth cost of enterprises; by using a lightweight model to perform super-resolution, real-time super-resolution can be achieved on mobile terminals; in addition, by comparing the high-frequency and low-frequency pictures generated from a low-definition picture with the high-frequency and low-frequency pictures extracted from a high-definition image using wavelet transform to calculate the loss function, the prior knowledge of edge extraction by wavelet decomposition can be used to reduce the complexity of super-resolution model convolution, and at the same time, the super-resolution processing process can pay more attention to high-frequency information, effectively improving the clarity of the super-resolved image in the human eye. See the following for details.

[0033] The following takes the mobile terminal as the execution subject and introduces the technical solution of the present application through multiple embodiments. It should be noted that these embodiments can be implemented in many different forms and should not be construed as being limited only to the embodiments described herein.

[0034] Embodiment 1 Figure 1 The flowchart of the image processing method according to Embodiment 1 of the present application is schematically shown.

[0035] As Figure 1 shown, the image processing method may include steps S100 to S106, where: Step S100, decomposing the target image to be super-resolved into a first Y-channel image and a first UV-channel image.

[0036] Step S102, generating a plurality of first frequency maps of the first Y-channel image based on the first Y-channel image using a trained lightweight model, and the first frequency maps include high-frequency frequency maps and low-frequency frequency maps.

[0037] Step S104: Based on the first frequency map, perform an inverse wavelet transform to obtain the high-definition first Y-channel map corresponding to the first Y-channel map.

[0038] Step S106: Based on the high-definition first Y-channel map and the first UV-channel map, obtain the high-definition image of the target image.

[0039] The image processing method provided in this embodiment decomposes the target image to be super-resolved into a first Y-channel map and a first UV-channel map, generates several first frequency maps of the first Y-channel map based on the first Y-channel map using a trained lightweight model, performs an inverse wavelet transform based on the first frequency map to obtain the high-definition first Y-channel map corresponding to the first Y-channel map, and obtains the high-definition image of the target image based on the high-definition first Y-channel map and the first UV-channel map. It can perform super-resolution processing of the image on a mobile terminal, reducing the enterprise's bandwidth cost; by using a lightweight model to perform super-resolution and only processing the Y-channel map to generate frequency maps, it can effectively reduce the occupation of memory and computing resources during the super-resolution processing, enabling the mobile terminal to achieve real-time super-resolution; at the same time, by using wavelet transform during the process, the super-resolution processing process can pay more attention to the information in the high-frequency aspect, thereby improving the clarity of the contour edges in the image and effectively improving the clarity of the super-resolved image to the human eye.

[0040] The following Figure 1 will elaborate in detail on each step in steps S100 to S106 and other optional steps.

[0041] Step S100 Decompose the target image to be super-resolved into a first Y-channel map and a first UV-channel map.

[0042] The target image to be super-resolved can be an image with a low resolution, including ordinary pictures and frame images in videos.

[0043] Specifically, the target image to be super-resolved can be decoded into bitmap data, and then the target image is decomposed into a first Y-channel map and a first UV-channel map through a shader. For example, the mobile terminal downloads a low-resolution picture, decodes the low-resolution picture into bitmap data, and then decomposes the bitmap data into a Y-channel map and a UV-channel map through a shader. In addition, in addition to the shader, an image processing library can also be used to decompose the target image into a first Y-channel map and a first UV-channel map.

[0044] Step S102 Based on the first Y-channel map, use a trained lightweight model to generate several first frequency maps of the first Y-channel map. The first frequency maps include high-frequency frequency maps and low-frequency frequency maps.

[0045] The lightweight model can be a convolutional neural network with fewer layers, such as a convolutional neural network with about 10 layers, so as to reduce the memory and computing resources occupied by the mobile terminal when the model is executed. Optionally, it can also be a convolutional neural network plus an attention mechanism, which can improve the super-resolution effect, but will increase the amount of computation at the same time. To reduce the amount of computation, the convolutional neural network can also use a smaller convolutional kernel, such as a 3*3 convolutional kernel.

[0046] In an alternative embodiment, the first frequency map includes a low-frequency frequency map (LL), a horizontal high-frequency frequency map (LH), a vertical high-frequency frequency map (HL), and a diagonal high-frequency frequency map (HH). When generating several first frequency maps of the first Y-channel map based on the trained lightweight model using the first Y-channel map, the number of first frequency maps can be a multiple of 4. For example, the first frequency map can correspond to the low-frequency frequency map, horizontal high-frequency frequency map, vertical high-frequency frequency map, and diagonal high-frequency frequency map of single-layer decomposition. At this time, the number of first frequency maps is 4; it can also correspond to the frequency maps of two-layer decomposition, that is, the low-frequency frequency map of single-layer decomposition can be further decomposed into a low-frequency frequency map, a horizontal high-frequency frequency map, a vertical high-frequency frequency map, and a diagonal high-frequency frequency map. At this time, the number of first frequency maps is 8. Correspondingly, when the number of first frequency maps is 4, a 2-fold magnification is achieved; when the number of first frequency maps is 8, a 4-fold magnification is achieved, and other quantities can be deduced by analogy.

[0047] In an alternative embodiment, the lightweight model includes several layers of convolution and residual connections.

[0048] Please refer to Figure 2 , which is a schematic diagram of the principle architecture of the image processing method. As shown in the figure, the lightweight model can include a shallow feature extraction layer and several deep feature extraction layers. The lightweight model can also include residual connections, directly adding the output of the shallow feature extraction layer to the output of the subsequent deep feature extraction layer, thereby solving the problem of gradient disappearance in the lightweight model.

[0049] It should be understood that although the number of feature extraction layers included in the lightweight model is less than that of a normal model, the problem of gradient disappearance will still exist. Adding residual connections can effectively solve the problem of gradient disappearance; moreover, since the number of feature extraction layers included in the lightweight model is small and the feature extraction ability is relatively lacking, using residual connections can make up for the lack of network depth and effectively improve the feature extraction ability, thereby improving the super-resolution effect while improving the real-time super-resolution of the mobile terminal.

[0050] In an alternative embodiment, as Figure 3 shown, the image processing method of the embodiment of the present application may further include: Step S200, degrading the high-definition image dataset to obtain a low-definition image dataset.

[0051] Step S202: Decompose each high-definition image in the high-definition image dataset into a second Y-channel map and a second UV-channel map.

[0052] Step S204: Decompose the second Y-channel map into several second frequency maps through wavelet transform.

[0053] Step S206: Decompose each low-definition image in the low-definition image dataset into a third Y-channel map and a third UV-channel map.

[0054] Step S208: Generate several third frequency maps of the third Y-channel map based on the third Y-channel map using a lightweight model.

[0055] Step S210: Train the lightweight model based on the second frequency maps and the corresponding third frequency maps of the third Y-channel map to obtain a trained lightweight model.

[0056] The high-definition image dataset can include multiple high-clarity images, which can include ordinary high-definition pictures and frame images extracted from high-definition videos.

[0057] Specifically, degradation methods such as lossy compression, blurring, and adding noise can be used to obtain the low-definition image dataset corresponding to the high-definition image dataset. For example, the corresponding low-definition image dataset can be obtained by compressing the high-definition image dataset to 1 / 2. For the high-definition image dataset, each high-definition image can be decomposed into a second Y-channel map and a second UV-channel map, and then the second Y-channel map can be decomposed into several second frequency maps through wavelet transform. Among them, the second frequency maps can include low-frequency maps, horizontal high-frequency maps, vertical high-frequency maps, and diagonal high-frequency maps, and the number of second frequency maps can also be a multiple of 4. Please refer to Figure 4 , which is an example diagram of decomposing a high-definition image into high and low frequency images. As shown in the figure, the high-definition image can be decomposed to obtain the corresponding low-frequency map, horizontal high-frequency map, vertical high-frequency map, and diagonal high-frequency map. In the figure, to protect the privacy of the person, the eyes of the person in the figure are blurred. For the low-definition image dataset, each low-definition image can be decomposed into a third Y-channel map and a third UV-channel map, and then a lightweight model is used to generate several third frequency maps of the third Y-channel map according to the third Y-channel map. After obtaining the second frequency maps and the third frequency maps, the second frequency maps can be compared with the corresponding third frequency maps to calculate the loss function, and the lightweight model is trained according to the calculation of the loss function. After the accuracy of the lightweight model reaches the preset accuracy, a trained lightweight model is obtained.

[0058] In this embodiment, the lightweight model is trained by comparing the high-frequency and low-frequency images obtained by decomposing the high-definition image through wavelet transform with the high-frequency and low-frequency images generated by the lightweight model corresponding to the low-definition image, so that the lightweight model can effectively generate the corresponding high-frequency and low-frequency images according to the low-definition image, and thus obtain the final high-definition image to realize the super-resolution processing of the image. Since the frequency map includes three high-frequency information and one low-frequency information, the trained lightweight model can pay more attention to the information in the high-frequency aspect, so that the contour edges of the super-resolved image are clearer and the visual perception of the super-resolved image by the human eye is improved. At the same time, by decomposing the image into the Y channel and the UV channel and only processing the Y channel, the computational amount in the super-resolution processing process can be reduced, and the super-resolution process of the mobile terminal can achieve the effect of real-time processing.

[0059] In an alternative embodiment, as Figure 5 shown, in step S210, training the lightweight model based on the second frequency map and the third frequency map corresponding to the second frequency map may include: Step S300, determining the peak signal-to-noise ratio of the second frequency map and the corresponding third frequency map as the first loss function.

[0060] Step S302, determining the structural similarity index of the second frequency map and the corresponding third frequency map as the second loss function.

[0061] Step S304, performing weighted summation based on the first loss function and the second loss function as the total loss function of the lightweight model, and training the lightweight model based on the total loss function.

[0062] For example, the second frequency maps obtained by wavelet decomposition of the high-definition image are respectively the low-frequency frequency map A, the horizontal high-frequency frequency map A, the vertical high-frequency frequency map A, and the diagonal high-frequency frequency map A, and the third frequency maps generated by the lightweight model corresponding to the low-definition image are respectively the low-frequency frequency map B, the horizontal high-frequency frequency map B, the vertical high-frequency frequency map B, and the diagonal high-frequency frequency map B. Then, the peak signal-to-noise ratios of the low-frequency frequency map A and the low-frequency frequency map B, the horizontal high-frequency frequency map A and the horizontal high-frequency frequency map B, the vertical high-frequency frequency map A and the vertical high-frequency frequency map B, and the diagonal high-frequency frequency map A and the diagonal high-frequency frequency map B can be calculated as the first loss function, and their structural similarity indices can be used as the second loss function. The first loss function and the second loss function are weighted and summed as the total loss function to train the lightweight model.

[0063] In this embodiment, by calculating the peak signal-to-noise ratio and structural similarity index between the high and low frequency images obtained by decomposing the high-definition image through wavelet transform and the high and low frequency images generated by the lightweight model from the corresponding low-definition image, and performing weighted summation as the total loss function to train the lightweight model, the trained lightweight model can effectively generate the high and low frequency images corresponding to the high-definition image according to the low-definition image. At the same time, the lightweight model can also pay more attention to the high-frequency information of the image, so that the contour edges of the super-resolved image are clearer, and the clarity of the super-resolved image in the human eye is improved.

[0064] In an alternative embodiment, the image processing method of the embodiments of the present application further includes: converting the trained lightweight model into a first model file and a second model file, where the first model file is used to run on an Android mobile terminal, and the second model file is used to run on an iOS mobile terminal.

[0065] Specifically, after the lightweight model is trained, the trained lightweight model is converted into a tflite model file for running on a mobile terminal with an Android operating system and a CoreML model file for running on a mobile terminal with an iOS operating system, so that the trained lightweight model can run on mobile terminals with these two operating systems to achieve super-resolution processing.

[0066] Step S104 , based on the first frequency map, the high-definition first Y-channel map corresponding to the first Y-channel map is obtained by inverse wavelet transform.

[0067] Please continue to refer to Figure 2 , after generating the low-frequency frequency map, horizontal high-frequency frequency map, vertical high-frequency frequency map, and diagonal high-frequency frequency map of the first Y-channel map, the image reconstruction module generates the high-definition first Y-channel map corresponding to the first Y-channel map according to these 4 frequency maps. Among them, the image reconstruction module inverse-transforms the first frequency map into the high-definition first Y-channel map through inverse wavelet transform.

[0068] Step S106 , based on the high-definition first Y-channel map and the first UV-channel map, the high-definition image of the target image is obtained.

[0069] Since the high-definition first Y-channel map is twice the size of the first Y-channel map, the first UV-channel map can be first enlarged to twice its size and then combined with the high-definition first Y-channel map to form the high-definition image of the target image. When the first frequency map is other multiples of 4, similar processing can be performed to form the high-definition image of the target image. For example, when the first frequency map is 8, the high-definition first Y-channel map is 4 times the size of the first Y-channel map, then the first UV-channel map can be first enlarged to 4 times its size and then combined with the high-definition first Y-channel map to form the high-definition image of the target image.

[0070] Embodiment 2 Figure 6 Schematically shown is a block diagram of an image processing apparatus according to Embodiment 2 of the present application. The apparatus can be divided into one or more program modules. One or more program modules are stored in a storage medium and executed by one or more processors to complete the embodiments of the present application. The program modules referred to in the embodiments of the present application refer to a series of computer program instruction segments that can complete specific functions. The following description will specifically introduce the functions of each program module in this embodiment. As Figure 6 shown, the apparatus 400 may include a decomposition module 410, a generation module 420, a transformation module 430, and a synthesis module 440, where: The decomposition module 410 is configured to decompose a target image to be super-resolved into a first Y-channel map and a first UV-channel map; The generation module 420 is configured to generate a plurality of first frequency maps of the first Y-channel map based on the first Y-channel map using a trained lightweight model, where the first frequency maps include high-frequency maps and low-frequency maps; The transformation module 430 is configured to obtain a high-definition first Y-channel map corresponding to the first Y-channel map by performing an inverse wavelet transform based on the first frequency map; The synthesis module 440 is configured to obtain a high-definition image of the target image based on the high-definition first Y-channel map and the first UV-channel map.

[0071] In an alternative embodiment, the first frequency maps include a low-frequency map, a horizontal high-frequency map, a vertical high-frequency map, and a diagonal high-frequency map.

[0072] In an alternative embodiment, the apparatus 400 is further configured to: Degrade a high-definition image dataset to obtain a low-definition image dataset; Decompose each high-definition image in the high-definition image dataset into a second Y-channel map and a second UV-channel map; Decompose the second Y-channel map into a plurality of second frequency maps by performing a wavelet transform; Decompose each low-definition image in the low-definition image dataset into a third Y-channel map and a third UV-channel map; Generate a plurality of third frequency maps of the third Y-channel map based on the third Y-channel map using a lightweight model; Train the lightweight model based on the second frequency maps and the corresponding third frequency maps of the second frequency maps to obtain the trained lightweight model.

[0073] In an alternative embodiment, the apparatus 400 is further configured to: Determine the peak signal-to-noise ratio of the second frequency map and the corresponding third frequency map as a first loss function; Determine the structural similarity index between the second frequency map and the corresponding third frequency map as the second loss function; Perform weighted summation based on the first loss function and the second loss function as the total loss function of the lightweight model, and train the lightweight model based on the total loss function.

[0074] In an alternative embodiment, the apparatus 400 is further configured to: Convert the trained lightweight model into a first model file and a second model file, where the first model file is used to run on an Android mobile terminal, and the second model file is used to run on an iOS mobile terminal.

[0075] In an alternative embodiment, the lightweight model includes several layers of convolution and residual connections.

[0076] Embodiment III Figure 7 Schematically shows a hardware architecture diagram of a computer device 10000 suitable for implementing the image processing method according to Embodiment III of the present application. In some embodiments, the computer device 10000 may be a terminal device such as a smart phone, a wearable device, a tablet computer, a personal computer, a vehicle-mounted terminal, a game console, a virtual device, a workbench, a digital assistant, a set-top box, a robot, etc. In other embodiments, the computer device 10000 may be a rack server, a blade server, a tower server, or a cabinet server (including an independent server or a server cluster composed of multiple servers). As Figure 7 shown, the computer device 10000 includes, but is not limited to: a memory 10010, a processor 10020, and a network interface 10030 that can communicate with each other through a system bus. Among them: The memory 10010 includes at least one type of computer-readable storage medium. The readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (such as SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disc, etc. In some embodiments, the memory 10010 may be an internal storage module of the computer device 10000, such as the hard disk or memory of the computer device 10000. In other embodiments, the memory 10010 may also be an external storage device of the computer device 10000, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc. equipped on the computer device 10000. Of course, the memory 10010 may also include both the internal storage module and the external storage device of the computer device 10000. In this embodiment, the memory 10010 is generally used to store the operating system and various application software installed on the computer device 10000, such as the program code of the image processing method. In addition, the memory 10010 may also be used to temporarily store various data that have been output or will be output.

[0077] In some embodiments, the processor 10020 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other chip. The processor 10020 is generally used to control the overall operation of the computer device 10000, such as performing control and processing related to data interaction or communication with the computer device 10000. In this embodiment, the processor 10020 is used to run the program code stored in the memory 10010 or process data.

[0078] The network interface 10030 may include a wireless network interface or a wired network interface, which is generally used to establish a communication link between the computer device 10000 and other computer devices. For example, the network interface 10030 is used to connect the computer device 10000 to an external terminal through a network, and establish a data transmission channel and a communication link between the computer device 10000 and the external terminal. The network may be a wireless or wired network such as an enterprise intranet (Intranet), the Internet, the Global System of Mobile communication (GSM for short), Wideband Code Division Multiple Access (WCDMA for short), 4G network, 5G network, Bluetooth, Wi-Fi, etc.

[0079] It should be noted that Figure 7 Only the computer device with components 10010 - 10030 is shown, but it should be understood that it is not required to implement all the shown components, and more or fewer components can be alternatively implemented.

[0080] In this embodiment, the image processing method stored in the memory 10010 can also be divided into one or more program modules and executed by one or more processors (such as the processor 10020) to complete the embodiments of the present application.

[0081] Embodiment 4 The embodiments of the present application further provide a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the image processing method in the embodiments are implemented.

[0082] In this embodiment, the computer-readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (such as SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disc, etc. In some embodiments, the computer-readable storage medium may be an internal storage unit of the computer device, such as the hard disk or memory of the computer device. In other embodiments, the computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc., equipped on the computer device. Of course, the computer-readable storage medium may also include both the internal storage unit and the external storage device of the computer device. In this embodiment, the computer-readable storage medium is generally used to store the operating system installed on the computer device and various application software, such as the program code of the image processing method in the embodiment. In addition, the computer-readable storage medium can also be used to temporarily store various data that have been output or will be output.

[0083] Embodiment 5 The embodiment of the present application further provides a computer program product, including a computer program, which when executed by a processor implements the method in the above embodiment.

[0084] Obviously, those skilled in the art should understand that the above-mentioned modules or steps of the embodiments of the present application can be implemented by a general-purpose computer device. They can be concentrated on a single computer device or distributed on a network composed of multiple computer devices. Optionally, they can be implemented by program codes executable by the computer device, so that they can be stored in a storage device and executed by the computer device. And in some cases, the steps shown or described can be executed in a different order than here, or they can be separately made into individual integrated circuit modules, or multiple modules or steps among them can be made into a single integrated circuit module to implement. Thus, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0085] It should be noted that the above are only the preferred embodiments of the present application, and do not limit the patent protection scope of the present application. Any equivalent structure or equivalent process transformation made by using the content of the specification and drawings of the present application, or directly or indirectly applied in other related technical fields, shall be equally included in the patent protection scope of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Decomposing the target image to be super-resolved into a first Y channel image and a first UV channel image; Generate a plurality of first frequency maps of the first Y channel map using a trained lightweight model based on the first Y channel map, where the first frequency map includes a high-frequency map and a low-frequency map; Based on the first frequency map, an inverse wavelet transform is used to obtain a high-definition first Y channel map corresponding to the first Y channel map; A high-definition image of the target image is obtained based on the high-definition first Y channel image and the first UV channel image.

2. The method according to claim 1, characterized in that: The first frequency graph includes a low frequency graph, a horizontal high frequency graph, a vertical high frequency graph, and a diagonal high frequency graph.

3. The method according to claim 1, characterized in that The method further comprises: A low-definition image dataset is obtained by degrading the high-definition image dataset; Decomposing each high-definition image in the high-definition image data set into a second Y channel image and a second UV channel image; Decomposing the second Y channel graph into a plurality of second frequency graphs by wavelet transform; Decomposing each low-definition image in the low-definition image data set into a third Y channel image and a third UV channel image; Generate a plurality of third frequency graphs of the third Y channel graph using a lightweight model based on the third Y channel graph; The lightweight model is trained based on the second frequency map and the third frequency map corresponding to the second frequency map to obtain the trained lightweight model.

4. The method according to claim 3, characterized in that The training of the lightweight model based on the second frequency map and the third frequency map corresponding to the second frequency map includes: determining a peak signal-to-noise ratio of the second frequency map and the corresponding third frequency map as a first loss function; determining a structural similarity index between the second frequency map and the corresponding third frequency map as a second loss function; A weighted sum is performed based on the first loss function and the second loss function as a total loss function of the lightweight model, and the lightweight model is trained based on the total loss function.

5. The method according to claim 3, characterized in that: The method further comprises: The trained lightweight model is converted into a first model file and a second model file, wherein the first model file is used to run on an Android mobile terminal, and the second model file is used to run on an IOS mobile terminal.

6. The method according to any one of claims 1 to 5, characterized in that: The lightweight model includes several layers of convolution and residual connections.

7. An image processing device, characterized in that: The device comprises: A decomposition module, used for decomposing the target image to be super-resolved into a first Y channel image and a first UV channel image; A generating module, configured to generate a plurality of first frequency maps of the first Y channel map using a trained lightweight model based on the first Y channel map, wherein the first frequency map includes a high-frequency map and a low-frequency map; A transformation module, configured to obtain a high-definition first Y channel image corresponding to the first Y channel image by using an inverse wavelet transform based on the first frequency image; A synthesis module is used to obtain a high-definition image of the target image based on the high-definition first Y channel image and the first UV channel image.

8. A computer device, characterized in that: include: at least one processor; and a memory communicatively connected to the at least one processor; wherein: The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images