Image processing method and device, electronic equipment and model training method

Abstract

The application discloses an image processing method and device, an electronic equipment and a model training method. The method comprises the following steps: acquiring a first image to be converted; converting the first image into a second image according to an image super-resolution model, wherein the resolution of the second image is higher than that of the first image, the image super-resolution model is obtained by adjusting a generative adversarial network model using a target loss function, and the target loss function comprises a difference loss function and a discrete cosine transform loss function; and generating a video corresponding to the second image according to the second image. The application solves the technical problem of low accuracy of existing super-resolution algorithms in generating super-resolution images.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image recognition, in particular to an image processing method and device, an electronic device and a model training method. BACKGROUND

[0002] In current mainstream image and video software, storing images and videos of different resolutions will consume huge storage space and transmission bandwidth, causing an increase in cost. In order to facilitate storage and transmission, images and videos are usually compressed. Most current display devices use super-resolution algorithms to improve display effects when viewing images and videos, but current super-resolution algorithms often cannot achieve a balance between subjective perception and objective evaluation indicators, causing a trade-off between subjectivity and objectivity, resulting in low precision when generating super-resolution images.

[0003] At present, there is no effective solution to the above problems. SUMMARY

[0004] Embodiments of the present application provide an image processing method, device, electronic device and model training method to at least solve the technical problem of low precision when generating super-resolution images by current super-resolution algorithms.

[0005] According to an aspect of an embodiment of the present application, an image processing method is provided, comprising: obtaining a first image to be converted; converting the first image into a second image according to an image super-resolution model, wherein the resolution of the second image is higher than that of the first image, and the image super-resolution model is obtained by adjusting a generative adversarial network model using a target loss function, and the target loss function includes a difference loss function and a discrete cosine transform loss function; generating a video corresponding to the second image according to the second image.

[0006] Optionally, obtaining the first image to be converted comprises: obtaining first video data to be converted, wherein the first video data is obtained by compressing high-resolution video data, and the high-resolution video data is composed of high-resolution images; and the first video data is cut according to a preset manner, and a cutting number is marked for the cut first image, wherein the cutting number is used to represent the cutting order of the first image.

[0007] Optionally, the image super-resolution model is obtained by the following method: obtaining a training image, wherein the training image is a low-resolution image; inputting the training image into a generative adversarial network model for training, and replacing the loss function of the generative adversarial network model with a target loss function; adjusting the parameters of the generative adversarial network model according to the target loss function to obtain the image super-resolution model.

[0008] Optionally, the differential loss function is determined by: obtaining the difference between the pixels of the second image generated by the image super-resolution model and the high-resolution image in terms of image height and image width respectively; determining the differential loss function based on at least the difference, the height and width of the super-resolution image, wherein the height and width of the second image and the high-resolution image are the same.

[0009] Optionally, the discrete cosine transform loss function is determined by: performing a discrete cosine transform on the high-resolution image to obtain a first transform value; performing a discrete cosine transform on the second image generated by the image super-resolution model to obtain a second transform value; and determining the discrete cosine transform loss function based on at least the first transform value, the second transform value, and the width and height of the second image.

[0010] Optionally, based on the second image, a video corresponding to the second image is generated, including: obtaining the second image and the interception number corresponding to the second image, wherein the interception number corresponding to the second image is the interception number marked by the first image before the second image is converted; splicing the second image according to the interception number to obtain a second video corresponding to the second image.

[0011] According to another aspect of an embodiment of the present application, a model training method is also provided, including: obtaining a sample image; inputting the sample image into a generative adversarial network model for training to obtain a predicted image, wherein the resolution of the predicted image is higher than the resolution of the sample image; using a differential loss function to determine a first difference between a real image and a predicted image of the sample image, and using a discrete cosine transform loss function to determine a second difference between the real image and the predicted image of the sample image; and adjusting the model parameters of the generative adversarial network model based on the first difference and the second difference.

[0012] Optionally, the differential loss function is determined by: obtaining the difference between the pixels of the predicted image and the real image of the sample image in image height and image width respectively; determining the differential loss function based on at least the difference, the height and width of the predicted image, wherein the height and width of the real image of the predicted image and the sample image are the same.

[0013] Optionally, the discrete cosine transform loss function is determined by: performing a discrete cosine transform on the real image of the sample image to obtain a first transform value; performing a discrete cosine transform on the predicted image to obtain a second transform value; and determining the discrete cosine transform loss function based on at least the first transform value, the second transform value, and the width and height of the predicted image.

[0014] According to another aspect of the embodiments of the present application, an image processing device is also provided, including: an acquisition module for acquiring a first image to be converted; a conversion module for converting the first image into a second image based on an image super-resolution model, wherein the resolution of the second image is higher than the resolution of the first image, and the image super-resolution model is obtained by adjusting the generative adversarial network model using a target loss function, and the target loss function includes a differential loss function and a discrete cosine transform loss function; a generation module for generating a video corresponding to the second image based on the second image.

[0015] According to another aspect of an embodiment of the present application, an electronic device is also provided, including: a memory and a processor, the processor being used to run a computer program stored in the memory, wherein the computer program executes the above-mentioned image processing method or the above-mentioned model training method when running.

[0016] According to another aspect of an embodiment of the present application, a non-volatile storage medium is also provided, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the above-mentioned image processing method or the above-mentioned model training method by running the computer program.

[0017] In an embodiment of the present application, a first image to be converted is obtained; the first image is converted into a second image according to an image super-resolution model, wherein the resolution of the second image is higher than the resolution of the first image, and the image super-resolution model is obtained by adjusting the generative adversarial network model using a target loss function, and the target loss function includes a differential loss function and a discrete cosine transform loss function; based on the second image, a video corresponding to the second image is generated, thereby achieving the purpose of using the target loss function to adjust the generative adversarial network to obtain the image super-resolution model, thereby achieving the technical effect of enhancing the edge texture feature information of objects in the image, and further solving the technical problem of low precision of existing super-resolution algorithms when generating super-resolution images. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0019] Figure 1 This is a hardware structure block diagram of a computer terminal (or electronic device) for implementing an image processing method or a model training method according to an embodiment of the present application;

[0020] Figure 2 is a flowchart of an image processing method according to an embodiment of the present application;

[0021] Figure 3This is an effect diagram of an image processed by a different model according to an embodiment of the present application;

[0022] Figure 4 is a flow chart of a model training method according to an embodiment of the present application;

[0023] Figure 5 is a structural diagram of an image processing device according to an embodiment of the present application;

[0024] Figure 6 is a structural diagram of a model training device according to an embodiment of the present application;

[0025] Figure 7 This is a flowchart of converting a low-resolution video into a super-resolution video according to an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] The image processing method or model training method provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 The hardware structure block diagram of a computer terminal (or electronic device) for implementing an image processing method or a model training method is shown. Figure 1As shown, the computer terminal 10 (or electronic device 10) may include one or more (illustrated by 102a, 102b, ..., 102n in the figure) processors (the processor may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission module 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a power supply and / or a camera. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0029] It should be noted that the one or more processors and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry." The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuitry may be a single, independent processing module, or may be incorporated in whole or in part into any of the other components of the computer terminal 10 (or electronic device). As described in the embodiments of the present application, the data processing circuitry serves as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0030] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the image processing method or model training method in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, realizing the above-mentioned image processing method or model training method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor, and these remote memories can be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0031] The transmission module 106 is configured to receive or transmit data via a network. A specific example of the aforementioned network may include a wireless network provided by the communications provider of the computer terminal 10. In one embodiment, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission device 106 may be a radio frequency (RF) module, which is configured to communicate with the Internet wirelessly.

[0032] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or electronic device).

[0033] It should be noted that, in some optional embodiments, the above Figure 1 The computer device (or electronic device) shown may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of hardware elements and software elements. Figure 1 This is merely one example of a particular embodiment and is intended to illustrate the types of components that may be present in the computer device (or electronic device) described above.

[0034] In the above-mentioned operating environment, the embodiments of the present application provide an embodiment of an image processing method and a model training method. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0035] Figure 2 is a flow chart of an image processing method according to an embodiment of the present application, such as Figure 2 As shown, the method includes the following steps:

[0036] Step S202, obtaining a first image to be converted;

[0037] Step S204: converting the first image into a second image according to an image super-resolution model, wherein the resolution of the second image is higher than the resolution of the first image, wherein the image super-resolution model is obtained by adjusting the generative adversarial network model using a target loss function, wherein the target loss function includes a differential loss function and a discrete cosine transform loss function;

[0038] Step S206: Generate a video corresponding to the second image based on the second image.

[0039] In the above steps S202 to S206, the first image can be a low-resolution image, and the second image can be a super-resolution image. The original generative adversarial network model is adjusted using the above-mentioned target loss function to obtain an image super-resolution model. The low-resolution image is converted into a high-resolution image using the trained image super-resolution model, which can make the display effect of the super-resolution image obtained after conversion better and bring better subjective visual experience to the user.

[0040] In step S202 of the above-mentioned image processing method, obtaining the first image to be converted specifically includes the following steps: obtaining the first video data to be converted, wherein the first video data is obtained by compressing high-resolution video data, and the high-resolution video data is composed of high-resolution images; intercepting the first video data according to a preset method, and marking the intercepted first image with an interception number, wherein the interception number is used to represent the interception order of the first image.

[0041] In an embodiment of the present application, in order to facilitate the storage of images or videos, the original high-resolution image or high-resolution video data is usually compressed to obtain compressed low-resolution image or low-resolution video data. When the display device displays these compressed low-resolution images or low-resolution video data, in order to improve the display effect, a super-resolution algorithm is usually used to convert the low-resolution image into a super-resolution image.

[0042] When low-resolution video data needs to be converted, the low-resolution video data to be converted, that is, the above-mentioned first video data, needs to be obtained first. By intercepting the low-resolution video data according to a preset method, at least one low-resolution image corresponding to the low-resolution video data can be obtained. The preset method can be frame-by-frame interception, for example, intercepting a low-resolution image every other frame, or intercepting according to odd frames and even frames, etc. The interception method can be pre-set, and the specific interception method is not limited in the embodiment of the present application.

[0043] After the low-resolution video data is captured according to a predetermined method, the resulting low-resolution images need to be labeled with a capture number. This low-resolution image is also referred to as the first image. For illustration, taking the example of capturing a low-resolution image every other frame, multiple low-resolution images are generated after the low-resolution video data is captured frame by frame. These low-resolution images are labeled with capture numbers, such as Image 1, Image 2, ..., Image n. These capture numbers indicate the order in which the low-resolution images were captured.

[0044] In step S204 of the image processing method, the image super-resolution model is trained by the following method: obtaining a training image, wherein the training image is a low-resolution image; inputting the training image into a generative adversarial network model for training, and replacing a loss function of the generative adversarial network model with a target loss function; adjusting parameters of the generative adversarial network model according to the target loss function to obtain the image super-resolution model.

[0045] In the embodiment, the original generative adversarial network model is trained by using a low-resolution image, and a loss function in the generative adversarial network model is replaced with a difference loss function and a discrete cosine transform loss function, which are the target loss function. The difference between an image generated by the generative adversarial network model and a high-resolution image is calculated by the target loss function, and the generative adversarial network model is adjusted. After the adjustment is completed, the image super-resolution model is obtained. It should be noted that the high-resolution image is an image in high-resolution video data corresponding to the low-resolution video data before compression, and the image super-resolution model is used to restore the compressed low-resolution image to the original high-resolution image. The image super-resolution model obtained by adjusting the target loss function has a better restoration effect.

[0046] In step S204 of the image processing method, the difference loss function is determined by the following method: obtaining a difference value between pixels of a second image generated by the image super-resolution model and a high-resolution image in image height and image width; and determining the difference loss function according to at least the difference value, the height and the width of the super-resolution image, wherein the height and the width of the second image and the high-resolution image are the same.

[0047] In the embodiment, the difference loss function can be calculated by the following formula:

[0048]

[0049] In the formula, l d represents the difference loss, W and H are the width and height of the image, I HR is a pixel value of a high-resolution image corresponding to the low-resolution image (i.e., the first image) before compression, I SR is a pixel value of a super-resolution image (i.e., the second image) generated by the model, is a pixel value of the high-resolution image in the width, is a pixel value of the super-resolution image in the width, is a pixel value of the high-resolution image in the height, is a pixel value of the super-resolution image in the height, d x represents the differential in the x-axis direction, dy It should be noted that the height and width of the super-resolution image, high-resolution image, and low-resolution image corresponding to the same image are the same.

[0050] In step S204 of the above-mentioned image processing method, the discrete cosine transform loss function is determined in the following manner: performing a discrete cosine transform on the high-resolution image to obtain a first transform value; performing a discrete cosine transform on the second image generated by the image super-resolution model to obtain a second transform value; and determining the discrete cosine transform loss function based on at least the first transform value, the second transform value, and the width and height of the second image.

[0051] In the embodiment of the present application, the discrete cosine transform loss function can be calculated using the following formula:

[0052]

[0053] In the above formula, l dct Denotes discrete cosine transform loss, and DCT denotes discrete cosine transform function.

[0054] The parameters of the original generative adversarial network model are adjusted according to the above-mentioned differential loss function and discrete cosine transform loss function. During the model training process, the model parameters are adjusted only when both loss functions are satisfied at the same time. Using the differential loss function and discrete cosine transform loss function to adjust the generative adversarial network model can effectively enhance the edge texture feature information of objects in images or videos. Figure 3 The effect diagram of image processing for different models, Figure 3 In the above example, HR represents high-resolution images, and Ours represents the image super-resolution model obtained in this application. Figure 3 It can be seen from the figure that the super-resolution image generated by the image super-resolution model obtained in the embodiment of the present application has almost no artifacts, achieving a better visual effect overall, and also has good results in objective evaluation indicators (such as PSNR indicator and SSIM indicator). Figure 3 In the figure, the image super-resolution model obtained in this application has a PSNR index of 22.23 and a SSIM index of 0.49, while other models, such as the ESRGAN model, have a PSNR index of 21.48 and a SSIM index of 0.46. Therefore, the image super-resolution model obtained in this application has a better visual effect.

[0055] In step S206 of the above-mentioned image processing method, a video corresponding to the second image is generated based on the second image, which specifically includes the following steps: obtaining the second image and the interception number corresponding to the second image, wherein the interception number corresponding to the second image is the interception number marked by the first image before the second image is converted; splicing the second image according to the interception number to obtain a second video corresponding to the second image.

[0056] In an embodiment of the present application, after a low-resolution image is converted into a super-resolution image (i.e., a second image) using an image super-resolution model, the converted super-resolution image also has a cut-off number, and the cut-off number of the super-resolution image is the same as the cut-off number of the low-resolution image before the conversion. The order of the super-resolution images can be determined according to the cut-off number, and the converted super-resolution images can be spliced ​​into a super-resolution video according to the order.

[0057] Figure 4 is a flow chart of a model training method according to an embodiment of the present application, such as Figure 4 As shown, the method includes:

[0058] Step S402, obtaining a sample image;

[0059] Step S404: inputting the sample image into a generative adversarial network model for training to obtain a predicted image, wherein the resolution of the predicted image is higher than the resolution of the sample image;

[0060] Step S406, using a differential loss function to determine a first difference between the real image and the predicted image of the sample image, and using a discrete cosine transform loss function to determine a second difference between the real image and the predicted image of the sample image;

[0061] Step S408: Adjust model parameters of the generative adversarial network model according to the first difference and the second difference.

[0062] In step S406 of the above-mentioned model training method, the differential loss function is determined by: obtaining the difference between the pixels of the predicted image and the real image of the sample image in the image height and image width respectively; determining the differential loss function based on at least the difference, the height and width of the predicted image, wherein the height and width of the real image of the predicted image and the sample image are the same.

[0063] In step S406 of the above-mentioned model training method, the discrete cosine transform loss function is determined in the following manner: performing a discrete cosine transform on the real image of the sample image to obtain a first transform value; performing a discrete cosine transform on the predicted image to obtain a second transform value; and determining the discrete cosine transform loss function based on at least the first transform value, the second transform value, and the width and height of the predicted image.

[0064] It should be noted that Figure 4 The model training method shown can be used to train a generative adversarial network model, and then an image super-resolution model can be obtained. The relevant formulas of the differential loss function and the discrete cosine transform loss function are described above and will not be repeated here.

[0065] Figure 5 is a structural diagram of an image processing device according to an embodiment of the present application, such as Figure 5 As shown, the device includes:

[0066] An acquisition module 502 is configured to acquire a first image to be converted;

[0067] a conversion module 504 for converting the first image into a second image according to an image super-resolution model, wherein the resolution of the second image is higher than the resolution of the first image, wherein the image super-resolution model is obtained by adjusting the generative adversarial network model using a target loss function, wherein the target loss function includes a differential loss function and a discrete cosine transform loss function;

[0068] The generating module 506 is configured to generate a video corresponding to the second image based on the second image.

[0069] In the acquisition module in the above-mentioned image processing device, the acquisition module is also used to acquire the first video data to be converted, wherein the first video data is obtained by compressing high-resolution video data, and the high-resolution video data is composed of high-resolution images; the first video data is intercepted according to a preset method, and an interception number is marked for the intercepted first image, wherein the interception number is used to represent the interception order of the first image.

[0070] In the above-mentioned image processing device, the image super-resolution model is trained in the following manner: obtaining a training image, wherein the training image is a low-resolution image; inputting the training image into a generative adversarial network model for training, and replacing the loss function of the generative adversarial network model with a target loss function; adjusting the parameters of the generative adversarial network model according to the target loss function to obtain the image super-resolution model.

[0071] In the above-mentioned image processing device, the differential loss function is determined in the following manner: respectively obtaining the difference between the pixels of the second image generated by the image super-resolution model and the high-resolution image in terms of image height and image width; determining the differential loss function based on at least the difference, the height and width of the second image, wherein the height and width of the super-resolution image and the high-resolution image are the same.

[0072] In the above-mentioned image processing device, the discrete cosine transform loss function is determined in the following manner: performing a discrete cosine transform on the high-resolution image to obtain a first transform value; performing a discrete cosine transform on the second image generated by the image super-resolution model to obtain a second transform value; and determining the discrete cosine transform loss function based on at least the first transform value, the second transform value, and the width and height of the second image.

[0073] In the generation module in the above-mentioned image processing device, the generation module is also used to obtain a second image and a capture number corresponding to the second image, wherein the capture number corresponding to the second image is the capture number marked by the first image before the second image is converted; the second image is spliced ​​according to the capture number to obtain a second video corresponding to the second image.

[0074] It should be noted that Figure 5 The image processing apparatus shown is used to perform Figure 2 The image processing method shown, therefore Figure 2 The relevant explanations about the image processing method in are also applicable to the image processing device and will not be repeated here.

[0075] Figure 6 is a structural diagram of a model training device according to an embodiment of the present application, such as Figure 6 As shown, the device includes:

[0076] A first acquisition module 602 is used to acquire a sample image;

[0077] An input module 604 is configured to input the sample image into a generative adversarial network model for training to obtain a predicted image, wherein the resolution of the predicted image is higher than the resolution of the sample image;

[0078] a determination module 606 for determining a first difference between a true image and a predicted image of the sample image using a differential loss function, and determining a second difference between the true image and the predicted image of the sample image using a discrete cosine transform loss function;

[0079] The adjustment module 608 is configured to adjust model parameters of the generative adversarial network model according to the first difference and the second difference.

[0080] In the above-mentioned model training device, the differential loss function is determined in the following manner: respectively obtaining the difference between the pixels of the predicted image and the real image of the sample image in the image height and image width; determining the differential loss function based on at least the difference, the height and width of the predicted image, wherein the height and width of the real image of the predicted image and the sample image are the same.

[0081] In the above-mentioned model training device, the discrete cosine transform loss function is determined in the following manner: performing a discrete cosine transform on the real image of the sample image to obtain a first transform value; performing a discrete cosine transform on the predicted image to obtain a second transform value; and determining the discrete cosine transform loss function based on at least the first transform value, the second transform value, and the width and height of the predicted image.

[0082] It should be noted that Figure 6 The model training apparatus shown is used to perform Figure 4 The model training method shown, therefore the relevant explanations in the above model training method are also applicable to the model training device and will not be repeated here.

[0083] Figure 7 is a flow chart of converting a low-resolution video into a super-resolution video according to an embodiment of the present application. Figure 7 As shown, after a low-resolution video is captured in a preset manner, a low-resolution image is obtained. The low-resolution video is obtained by compressing the high-resolution video. The low-resolution image is input into a trained image super-resolution model to generate a super-resolution image. The generated super-resolution image is then spliced ​​to obtain a super-resolution video. During the training of the image super-resolution model, the training image (such as the low-resolution image) is input into the original generative adversarial network model for training to obtain a super-resolution image. The difference between the super-resolution image and the high-resolution image during training is continuously calculated using a difference loss function and a discrete cosine transform loss function. The parameters of the original generative adversarial network model are then updated based on the difference results, ultimately obtaining an image super-resolution model.

[0084] An embodiment of the present application also provides a non-volatile storage medium, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the following image processing method by running the computer program: obtaining a first image to be converted; converting the first image into a second image based on an image super-resolution model, wherein the resolution of the second image is higher than the resolution of the first image, and the image super-resolution model is obtained by adjusting a generative adversarial network model using a target loss function, and the target loss function includes a differential loss function and a discrete cosine transform loss function; based on the second image, generating a video corresponding to the second image.

[0085] The device where the above-mentioned non-volatile storage medium is located can also execute the following model training method by running a computer program: obtaining a sample image; inputting the sample image into a generative adversarial network model for training to obtain a predicted image, wherein the resolution of the predicted image is higher than the resolution of the sample image; using a differential loss function to determine a first difference between the real image and the predicted image of the sample image, and using a discrete cosine transform loss function to determine a second difference between the real image and the predicted image of the sample image; adjusting the model parameters of the generative adversarial network model based on the first difference and the second difference.

[0086] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0087] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0088] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of the units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0089] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0090] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0091] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0092] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. An image processing method, characterized in that: include: Acquire a first image to be converted, wherein the first image comprises an image with a cutout number cutout from low-resolution video data; Converting the first image into a second image according to an image super-resolution model, wherein the resolution of the second image is higher than the resolution of the first image, the image super-resolution model being obtained by adjusting a generative adversarial network model using a target loss function, the target loss function including a differential loss function and a discrete cosine transform loss function; generating a video corresponding to the second image based on the second image, wherein the video includes a video obtained by splicing the second images according to the interception numbers; Wherein, during the training process of the image super-resolution model, when the difference loss function and the discrete cosine transform loss function are satisfied at the same time, the parameters of the image super-resolution model are adjusted; the difference loss function is calculated by the following formula: Among them, l d represents the differential loss, W and H are the width and height of the image, I HR is the pixel value of the high-resolution image corresponding to the low-resolution image corresponding to the first image before compression, I SR is the pixel value of the super-resolution image corresponding to the second image generated by the model, is the pixel value of the high-resolution image in width, is the pixel value of the super-resolution image in width, is the pixel value of the high-resolution image at height, is the pixel value of the super-resolution image at height, dx represents the differential in the x-axis direction, and dy represents the differential in the y-axis direction.

2. The method according to claim 1, characterized in that Obtaining a first image to be converted, including: Acquire first video data to be converted, wherein the first video data is obtained by compressing high-resolution video data, and the high-resolution video data consists of high-resolution images; The first video data is intercepted according to a preset method, and an interception number is marked for the intercepted first image, wherein the interception number is used to represent the interception order of the first image.

3. The method according to claim 1, characterized in that The image super-resolution model is trained in the following way: Acquire a training image, wherein the training image is a low-resolution image; Inputting the training image into the generative adversarial network model for training, and replacing the loss function of the generative adversarial network model with the target loss function; The parameters of the generative adversarial network model are adjusted according to the target loss function to obtain the image super-resolution model.

4. The method according to claim 2, characterized in that The differential loss function is determined in the following way: Obtaining pixel differences between a second image generated by the image super-resolution model and the high-resolution image in terms of image height and image width, respectively; The differential loss function is determined based on at least the difference, a height and a width of the super-resolution image, wherein the second image and the high-resolution image have the same height and width.

5. The method according to claim 2, characterized in that The discrete cosine transform loss function is determined by: performing discrete cosine transform on the high-resolution image to obtain a first transform value; performing a discrete cosine transform on the second image generated by the image super-resolution model to obtain a second transformed value; The discrete cosine transform loss function is determined based on at least the first transform value, the second transform value, and the width and height of the second image.

6. The method according to claim 2, characterized in that Generating a video corresponding to the second image according to the second image includes: Acquire the second image and a cutout number corresponding to the second image, wherein the cutout number corresponding to the second image is the cutout number marked on the first image before the second image is converted; The second images are spliced ​​according to the interception numbers to obtain a second video corresponding to the second images.

7. A model training method, characterized in that: include: Acquire a sample image, wherein the sample image comprises an image with a capture number captured from low-resolution video data; Inputting the sample image into a generative adversarial network model for training to obtain a predicted image, wherein the resolution of the predicted image is higher than the resolution of the sample image; Determining a first difference between a real image of the sample image and the predicted image using a differential loss function, and determining a second difference between the real image of the sample image and the predicted image using a discrete cosine transform loss function; Adjusting model parameters of the generative adversarial network model according to the first difference and the second difference; Wherein, during the training process of the generative adversarial network model, when the differential loss function and the discrete cosine transform loss function are satisfied at the same time, the parameters of the generative adversarial network model are adjusted; the differential loss function is calculated by the following formula: Among them, l d Indicates the first difference, W and H are the width and height of the image, I HR is the pixel value of the high-resolution image corresponding to the low-resolution image corresponding to the sample image before compression, I SR is the pixel value of the super-resolution image generated by the model corresponding to the predicted image, is the pixel value of the high-resolution image in width, is the pixel value of the super-resolution image in width, is the pixel value of the high-resolution image at height, is the pixel value of the super-resolution image at height, dx represents the differential in the x-axis direction, and dy represents the differential in the y-axis direction.

8. The method according to claim 7, characterized in that The differential loss function is determined in the following way: respectively obtaining pixel differences between the predicted image and the real image of the sample image in terms of image height and image width; The differential loss function is determined based on at least the difference value, a height and a width of the predicted image, wherein the predicted image and a real image of the sample image have the same height and width.

9. The method according to claim 7, characterized in that The discrete cosine transform loss function is determined by: Performing discrete cosine transform on the real image of the sample image to obtain a first transform value; Performing discrete cosine transform on the predicted image to obtain a second transform value; The discrete cosine transform loss function is determined based on at least the first transform value, the second transform value, and the width and height of the predicted image.

10. An image processing device, characterized in that: include: An acquisition module, configured to acquire a first image to be converted, wherein the first image comprises an image with a capture number captured from low-resolution video data; A conversion module is configured to convert the first image into a second image according to an image super-resolution model, wherein the resolution of the second image is higher than that of the first image, and the image super-resolution model is obtained by adjusting the generative adversarial network model using a target loss function, wherein the target loss function includes a differential loss function and a discrete cosine transform loss function; during the training process of the image super-resolution model, when the differential loss function and the discrete cosine transform loss function are simultaneously satisfied, the parameters of the image super-resolution model are adjusted; the differential loss function is calculated by the following formula: Among them, l d represents the differential loss, W and H are the width and height of the image, I HR is the pixel value of the high-resolution image corresponding to the low-resolution image corresponding to the first image before compression, I SR is the pixel value of the super-resolution image corresponding to the second image generated by the model, is the pixel value of the high-resolution image in width, is the pixel value of the super-resolution image in width, is the pixel value of the high-resolution image at height, is the pixel value of the super-resolution image at height, dx represents the differential in the x-axis direction, and dy represents the differential in the y-axis direction; A generation module is used to generate a video corresponding to the second image based on the second image, wherein the video includes a video obtained by splicing the second images according to the interception numbers.

11. An electronic device, characterized in that: include: A memory and a processor, wherein the processor is used to run a computer program stored in the memory, wherein the computer program, when running, executes the image processing method described in any one of claims 1 to 6, or executes the model training method described in any one of claims 7 to 9.

12. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the image processing method described in any one of claims 1 to 6, or executes the model training method described in any one of claims 7 to 9 by running the computer program.

Images