Image processing method and device, electronic equipment, readable storage medium and program product

Abstract

The invention relates to an image processing method and device, electronic equipment, a readable storage medium and a program product. The method comprises the following steps: inputting original image data into a first image processing model to obtain first image data of a first image domain corresponding to the original image data; performing image domain conversion on the first image data to obtain second image data of a second image domain; the applicability of the artifact removal processing of the second image domain is higher than that of the artifact removal processing of the first image domain; performing artifact removal processing on the second image data to obtain third image data of a second image domain; and performing image domain inverse conversion on the third image data to obtain fourth image data of the first image domain. By adopting the method, artifacts can be removed more effectively, and an image more conforming to real scene distribution can be obtained.

Description

Technical Field

[0001] The present application relates to the field of camera technology, and in particular to an image processing method, device, electronic device, computer-readable storage medium, and computer program product. Background Art

[0002] With the development of image processing technology, various image processing models have emerged, which can automatically perform various image processing tasks, such as image restoration, image enhancement, image segmentation, and image classification. However, the relevant image processing models have the problem of artifacts that the processed images are easily inconsistent with natural scenes. Summary of the invention

[0003] The embodiments of the present application provide an image processing method, device, electronic device, and computer-readable storage medium, which can more effectively remove artifacts and obtain images that are more consistent with the distribution of real scenes.

[0004] In a first aspect, the present application provides an image processing method, comprising:

[0005] Inputting the original image data into a first image processing model to obtain first image data of a first image domain corresponding to the original image data;

[0006] Performing image domain conversion on the first image data to obtain second image data in a second image domain; the applicability of the artifact removal processing in the second image domain is higher than the applicability of the artifact removal processing in the first image domain;

[0007] performing artifact removal processing on the second image data to obtain third image data in the second image domain;

[0008] Perform image domain inverse conversion on the third image data to obtain fourth image data in the first image domain.

[0009] In one embodiment, performing artifact removal processing on the second image data to obtain third image data in the second image domain includes:

[0010] Obtaining a mask region of each semantic object in the second image data;

[0011] For each mask region of the semantic object, performing artifact removal processing on the mask region to obtain an artifact removal region;

[0012] Each artifact removal area is fused to obtain third image data of the second image domain.

[0013] In one embodiment, after obtaining the mask area of ​​each semantic object in the second image data, the method further includes:

[0014] For each mask region of the semantic object, obtaining a de-artifacting type of the mask region;

[0015] The step of performing artifact removal processing on the mask region of each semantic object to obtain the artifact removal region includes:

[0016] For each mask region of the semantic object, a de-artifacting algorithm corresponding to the de-artifacting type is used to perform de-artifacting processing on the mask region to obtain a de-artifacting region.

[0017] In one embodiment, before performing image domain conversion on the first image data to obtain second image data in a second image domain, the method further includes:

[0018] Obtaining processing operations required for artifact removal;

[0019] According to the processing operation, a second image domain is determined from at least two candidate image domains.

[0020] In one embodiment, the first image domain is an RGB domain, and the second image domain is an SRGB domain or a YUV domain; performing image domain conversion on the first image data to obtain second image data in the second image domain includes:

[0021] If the second image domain is an SRGB domain, converting the first image data in the RGB domain into second image data in the SRGB domain;

[0022] If the second image domain is a YUV domain, the first image data in the RGB domain is image-domain-converted into intermediate image data in the SRGB domain, and the intermediate image data in the SRGB domain is image-domain-converted into second image data in the YUV domain.

[0023] In one embodiment, the method further comprises:

[0024] Using the fourth image data and the original image data as a training data pair for artifact removal training; inputting the training data pair into a second image processing model, and performing image processing on the original image data through the second image processing model to obtain training image data;

[0025] The parameters of the second image processing model are adjusted according to the training image data and the fourth image data until a trained second image processing model is obtained when a training cutoff condition is met.

[0026] In a second aspect, the present application further provides an image processing device, comprising:

[0027] A model output acquisition module, used for inputting the original image data into the first image processing model to obtain first image data of the first image domain corresponding to the original image data;

[0028] An image domain conversion module, configured to perform image domain conversion on the first image data to obtain second image data in a second image domain; the applicability of the de-artifacting processing in the second image domain is higher than the applicability of the de-artifacting processing in the first image domain;

[0029] an artifact removal processing module, configured to perform artifact removal processing on the second image data to obtain third image data in the second image domain;

[0030] The image domain inverse conversion module is used to perform image domain inverse conversion on the third image data to obtain fourth image data in the first image domain.

[0031] In a third aspect, the present application further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0032] Inputting the original image data into a first image processing model to obtain first image data of a first image domain corresponding to the original image data;

[0033] Performing image domain conversion on the first image data to obtain second image data in a second image domain; the applicability of the artifact removal processing in the second image domain is higher than the applicability of the artifact removal processing in the first image domain;

[0034] performing artifact removal processing on the second image data to obtain third image data in the second image domain;

[0035] Perform image domain inverse conversion on the third image data to obtain fourth image data in the first image domain.

[0036] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0037] Inputting the original image data into a first image processing model to obtain first image data of a first image domain corresponding to the original image data;

[0038] Performing image domain conversion on the first image data to obtain second image data in a second image domain; the applicability of the artifact removal processing in the second image domain is higher than the applicability of the artifact removal processing in the first image domain;

[0039] performing artifact removal processing on the second image data to obtain third image data in the second image domain;

[0040] Perform image domain inverse conversion on the third image data to obtain fourth image data in the first image domain.

[0041] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0042] Inputting the original image data into a first image processing model to obtain first image data of a first image domain corresponding to the original image data;

[0043] Performing image domain conversion on the first image data to obtain second image data in a second image domain; the applicability of the artifact removal processing in the second image domain is higher than the applicability of the artifact removal processing in the first image domain;

[0044] performing artifact removal processing on the second image data to obtain third image data in the second image domain;

[0045] Perform image domain inverse conversion on the third image data to obtain fourth image data in the first image domain.

[0046] The above-mentioned image processing method, device, electronic device, computer-readable storage medium and computer program product input the original image data into the first image processing model, obtain the first image data of the first image domain corresponding to the original image data, perform image domain conversion on the first image data, obtain the second image data of the second image domain, the applicability of the de-artifacting processing in the second image domain is higher than the applicability of the de-artifacting processing in the first image domain, then, perform de-artifacting processing on the second image data in the second image domain, it is possible to obtain the third image data with effective de-artifacting, and then perform image domain inverse conversion on the third image data to obtain the fourth image data of the first image domain with effective de-artifacting, the fourth image data is more consistent with the real scene distribution, and the accuracy of image processing is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 is a schematic flow chart of an image processing method in one embodiment;

[0049] Figure 2 is a schematic flow chart of an image processing method in another embodiment;

[0050] Figure 3 is a schematic diagram of training a second image processing model in one embodiment;

[0051] Figure 4 An image output by a second image processing model before training in one embodiment;

[0052] Figure 5 An image output by a trained second image processing model in one embodiment;

[0053] Figure 6 is a structural block diagram of an image processing device in one embodiment;

[0054] Figure 7 FIG. 4 is a diagram showing the internal structure of an electronic device in one embodiment. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0056] In one embodiment, Figure 1 As shown, an image processing method is provided. This embodiment uses the method applied to an electronic device as an example for illustration. The electronic device may be a terminal or a server. It is understandable that the method may also be applied to a system including a terminal and a server, and implemented through the interaction between the terminal and the server. Among them, the terminal may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices may be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, projection devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The head-mounted devices may be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. The server may be an independent physical server, a server cluster or a distributed system consisting of multiple physical servers, or a cloud server that provides cloud computing services.

[0057] In this embodiment, the image processing method includes the following steps:

[0058] Step S102: inputting the original image data into a first image processing model to obtain first image data of a first image domain corresponding to the original image data.

[0059] The raw image data refers to data directly obtained from an image acquisition device (such as a digital camera, scanner, medical imaging device, etc.) without any compression, encoding or other substantial processing. The raw image data may be data in the raw image domain, such as the RAW domain.

[0060] The first image processing model is a system or framework that is built based on mathematical algorithms, statistical principles, computer science and technology, etc., and is used to analyze, process and understand images to achieve specific tasks. The first image processing model is a model with AI (Artificial Intelligence) capabilities, such as the convolutional neural network model (CNN), the recurrent neural network model (RNN) and its variants, or the generative adversarial network model (GAN).

[0061] The image domain refers to the space or category where the image is located, which defines the representation method, attributes and operations that can be performed on the image data. The first image domain is the image domain to which the image data output by the first image processing model belongs. The first image data is the image data output by the first image processing model. For example, the first image domain can be an RGB domain, and the first image data is RGB image data.

[0062] Optionally, the electronic device obtains the captured original image data, inputs the original image data into a first image processing model, and obtains first image data of a first image domain output by the first image processing model. The first image data carries artifacts.

[0063] Optionally, the electronic device obtains meta information of the original image data. Meta information refers to descriptive data related to the original image data. This data is not the pixel content of the image itself, but provides a lot of important information about the image, which helps to manage, understand and process the image. Exemplarily, the meta information of the original image data may include at least one of the image format, resolution, size, camera model, shooting time, aperture value, shutter speed and sensitivity.

[0064] The electronic device can obtain the meta information of the original image data from the shooting device to which the original image data belongs. The shooting device to which the original image data belongs can be the electronic device itself or another device. After the shooting device obtains the original image data by shooting, the original image data and the corresponding meta information are saved.

[0065] Step S104, performing image domain conversion on the first image data to obtain second image data in a second image domain; the applicability of the artifact removal processing in the second image domain is higher than the applicability of the artifact removal processing in the first image domain.

[0066] The second image domain is a target image domain obtained by converting the first image domain. The second image data is image data obtained after the first image data is converted through the image domain. For example, the second image domain may be an SRGB domain, and the second image data is SRGB image data. For another example, the second image domain may be a YUV domain, and the second image data is YUV image data.

[0067] It is understandable that the applicability of the first image domain is relatively poor. For example, it is difficult to directly perform algorithmic processing on image data in the RAW domain or image data in the linear RGB domain, and thus it is impossible to directly remove artifacts to construct training data. The applicability of the second image domain is relatively good. Many image processing algorithms and related operations are performed on the second image domain (such as the SRGB domain or the YUV domain). After the first image data is converted into the image domain, the image data can be more accurately de-artifacted in the second image domain to construct more accurate training data.

[0068] Optionally, the electronic device performs image domain conversion on the first image data according to the meta-information to obtain second image data in a second image domain.

[0069] The electronic device obtains relevant information of the first image data in the first image domain according to the meta-information, and converts the relevant information of the first image data in the first image domain to the second image domain according to a preset conversion algorithm between the first image domain and the second image domain, so as to obtain second image data in the second image domain.

[0070] Optionally, the electronic device preprocesses the first image data according to the meta information to obtain preprocessed first image data; and performs image domain conversion on the preprocessed first image data according to the meta information to obtain second image data in a second image domain. The preprocessing may include at least one of black level removal, AWB (Auto-White Balance), CCM (Color Correction Matrix), and gamma.

[0071] It is understandable that the electronic device first preprocesses the first image data, such as removing some noise data or performing enhancement processing, and then performs image domain conversion. Preprocessing is crucial for optimizing image quality, ensuring color accuracy and improving visual effects, and can provide better basic data for subsequent image domain conversion.

[0072] Step S106: performing artifact removal processing on the second image data to obtain third image data in the second image domain.

[0073] The third image data is image data obtained by removing artifacts from the second image data, and the third image data does not have artifacts.

[0074] Optionally, the electronic device determines a de-artifacting type of the second image data, and performs de-artifacting processing on the second image data using a de-artifacting algorithm corresponding to the de-artifacting type to obtain third image data in the second image domain.

[0075] Optionally, the electronic device performs artifact removal processing on each region in the second image data respectively to obtain each artifact removal region; and fuses each artifact removal region to obtain third image data in the second image domain.

[0076] Step S108: performing image domain inverse conversion on the third image data to obtain fourth image data in the first image domain.

[0077] The fourth image data is obtained by inversely transforming the third image data in the image domain. The fourth image data does not have artifacts.

[0078] Optionally, the electronic device performs image domain inverse conversion on the third image data according to the meta-information to obtain fourth image data in the first image domain.

[0079] The electronic device obtains relevant information of the third image data in the second image domain based on the meta-information, and converts the relevant information of the third image data in the second image domain to the first image domain according to a preset inverse conversion algorithm between the first image domain and the second image domain, thereby obtaining fourth image data in the first image domain.

[0080] The above-mentioned image processing method inputs the original image data into the first image processing model, obtains the first image data of the first image domain corresponding to the original image data, performs image domain conversion on the first image data, obtains the second image data of the second image domain, and the applicability of the de-artifacting processing in the second image domain is higher than the applicability of the de-artifacting processing in the first image domain. Then, the second image data in the second image domain is de-artifacted to obtain the third image data with effective de-artifacting, and then the third image data is inversely converted in the image domain to obtain the fourth image data of the first image domain with effective de-artifacting, and the fourth image data is more consistent with the real scene distribution, thereby improving the accuracy of image processing.

[0081] In one embodiment, the second image data is subjected to de-artifacting processing to obtain third image data in the second image domain, including: obtaining a mask area of ​​each semantic object in the second image data; for each mask area of ​​the semantic object, performing de-artifacting processing on the mask area to obtain a de-artifacted area; and fusing each de-artifacted area to obtain the third image data in the second image domain.

[0082] Among them, the second image data of the semantic object has a specific object or area with clear semantics that can be understood and distinguished. Exemplarily, the semantic objects in the second image data may include semantic objects such as vehicles, pedestrians, traffic signs, faces, buildings or the sky. The mask area is an image area obtained by masking the image to be processed (all or part) with a selected image, graphic or object. The mask area can be a two-dimensional matrix (or array) of the same size as the original image area, and its element values ​​are usually used to indicate whether the pixels at the corresponding position of the original image are selected or participate in subsequent processing.

[0083] Optionally, the electronic device performs semantic segmentation on the second image data, determines each semantic object in the second image data, and determines a mask area for each semantic object in the second image data.

[0084] Furthermore, the electronic device performs brightness detection and edge detection on the second image data to obtain detection information in the second image data, and performs semantic segmentation on the second image data based on the detection information to determine each semantic object in the second image data. It is understandable that the electronic device performs brightness detection and edge detection on the second image data to more accurately determine detection information such as different brightness and different edge contours of each different semantic object in the second image data, and then can more accurately semantically segment different semantic objects based on the detection information.

[0085] Optionally, after obtaining the mask area of ​​each semantic object in the second image data, it also includes: for the mask area of ​​each semantic object, obtaining a de-artifact type of the mask area; for the mask area of ​​each semantic object, performing de-artifact processing on the mask area to obtain a de-artifact area, including: for the mask area of ​​each semantic object, using a de-artifact algorithm corresponding to the de-artifact type to perform de-artifact processing on the mask area to obtain a de-artifact area.

[0086] It is understandable that different artifact removal types correspond to different artifact removal algorithms. Exemplarily, the artifact removal types may be ghost, worm noise, grid, horizontal and vertical stripes or color fringing, etc., and the corresponding artifact removal algorithms also correspond one to one.

[0087] Optionally, the electronic device marks the artifact removal type of the mask area in the mask area to which it belongs; for the mask area of ​​each semantic object, the artifact removal type marked in the mask area is obtained, the artifact removal algorithm corresponding to the artifact removal type is obtained, and the artifact removal algorithm is used to perform artifact removal processing on the mask area to obtain a artifact removal area.

[0088] Optionally, for each mask region of the semantic object, the electronic device detects the mask region and determines the artifact removal type of the mask region. Optionally, the electronic device may also obtain the artifact removal type of the mask region input manually.

[0089] Optionally, the electronic device splices each de-artifacted area to obtain the third image data of the second image domain. Optionally, the electronic device performs Laplacian pyramid fusion on each de-artifacted area to obtain the third image data of the second image domain. Optionally, the electronic device can also fuse each de-artifacted area in other ways, which are not limited here.

[0090] In this embodiment, the electronic device obtains the mask area of ​​each semantic object in the second image data, and then performs de-artifacting processing on the mask area of ​​each different semantic object respectively, so as to obtain each de-artifacting area more accurately, and then fuse them to obtain more accurate third image data.

[0091] Furthermore, for each mask area of ​​the semantic object, the de-artifacting type of the mask area is obtained, and then, the de-artifacting algorithm corresponding to the de-artifacting type is used to perform de-artifacting processing on the mask area in a targeted manner to obtain a de-artifacting area with more accurate de-artifacting effect and better effect.

[0092] In one embodiment, before performing image domain conversion on the first image data to obtain second image data in the second image domain, the method further includes: obtaining processing operations required for artifact removal processing; and determining the second image domain from at least two candidate image domains according to the processing operations.

[0093] The applicability of the artifact removal processing in the candidate image domains is higher than the applicability of the artifact removal processing in the first image domain. The at least two candidate image domains may include an SRGB domain and a YUV domain.

[0094] It is understandable that different image domains have different applicability to artifact removal processing, and the electronic device can select a corresponding more suitable second image domain according to the processing operation required for artifact removal, thereby more accurately performing artifact removal processing on the second image data.

[0095] Optionally, the electronic device obtains a correspondence between a processing operation and an image domain, matches the processing operation with the correspondence, and determines a second image domain from at least two candidate image domains.

[0096] Optionally, the first image domain is an RGB domain, and the second image domain is an SRGB domain or a YUV domain; performing image domain conversion on the first image data to obtain second image data in the second image domain includes: if the second image domain is the SRGB domain, image domain conversion of the first image data in the RGB domain into second image data in the SRGB domain; if the second image domain is the YUV domain, image domain conversion of the first image data in the RGB domain into intermediate image data in the SRGB domain, and image domain conversion of the intermediate image data in the SRGB domain into second image data in the YUV domain.

[0097] Among them, the RGB domain is a color representation method based on optical principles and is widely used in many fields such as electronic device display and image processing. The SRGB domain is a universal RGB color space standard, jointly developed by HP and Microsoft in 1996, to provide a unified color standard for computer monitors, scanners, printers and other devices, and solve the problem of inconsistent color display between different devices. The YUV domain is a way of color encoding, which decomposes color information into brightness (Y) and chroma (U, V) components; Y represents brightness, that is, grayscale value, which reflects the black and white information of the image, is an important part of visual perception, and determines the overall brightness of the image; U and V represent chroma, which is used to describe color and saturation, that is, the type and depth of color.

[0098] Optionally, if the second image domain is an SRGB domain, the first image data in the RGB domain is converted into second image data in the SRGB domain according to the meta-information; if the second image domain is a YUV domain, the first image data in the RGB domain is converted into intermediate image data in the SRGB domain according to the meta-information, and the intermediate image data in the SRGB domain is converted into second image data in the YUV domain.

[0099] Optionally, if the second image domain is a YUV domain, the first image data in the RGB domain is converted into second image data in the YUV domain.

[0100] Optionally, if the second image domain is a YUV domain, the first image data in the RGB domain is converted into second image data in the YUV domain according to the meta information.

[0101] In this embodiment, the electronic device obtains the processing operation required for the artifact removal process, and according to the processing operation, the second image domain can be more accurately determined from at least two candidate image domains. Further, the image domain conversion is more accurately performed for the second image domain being the SRGB domain or the YUV domain.

[0102] In one embodiment, Figure 2As shown, the electronic device inputs the original image data into the first image processing model to obtain first image data of the first image domain corresponding to the original image data; performs image domain conversion on the first image data according to the meta-information of the original image data to obtain second image data of the second image domain; obtains the mask area of ​​each semantic object in the second image data, and for the mask area of ​​each semantic object, performs artifact removal processing on the mask area using the artifact removal algorithm corresponding to the artifact removal type to obtain the artifact removal area, and fuses the artifact removal areas to obtain the third image data of the second image domain; performs image domain inverse conversion on the third image data to obtain the fourth image data of the first image domain.

[0103] In one embodiment, the above method also includes: using the fourth image data and the original image data as a training data pair for artifact removal training; inputting the training data pair into a second image processing model, performing image processing on the original image data through the second image processing model to obtain training image data; adjusting parameters of the second image processing model according to the training image data and the fourth image data until a trained second image processing model is obtained when a training cutoff condition is met.

[0104] The second image processing model is a system or framework used for training to analyze, process and understand images to achieve a specific task. The second image processing model and the first image processing model may be the same or different.

[0105] The training cut-off condition can be set as needed. For example, the training cut-off condition can be one of the following: the number of training times reaches a preset number of times, the training duration reaches a preset duration, and the loss value between the training image data and the fourth image data is less than a preset threshold.

[0106] Optionally, the electronic device determines a loss value between the training image data and the fourth image data. If the loss value does not meet the training cutoff condition, the electronic device adjusts the parameters of the second image processing model and re-executes the step of inputting the training data into the second image processing model until the loss value meets the training cutoff condition and a trained second image processing model is obtained.

[0107] like Figure 3 As shown, the electronic device inputs the original image data into the second image processing model for image processing to obtain training image data, determines the loss value between the training image data and the fourth image data, and obtains a trained second image processing model if the loss value meets the training cutoff condition.

[0108] In this embodiment, the electronic device uses the fourth image data and the original image data as a training data pair for artifact removal training, inputs the training data pair into the second image processing model, performs image processing on the original image data through the second image processing model to obtain training image data, and adjusts the parameters of the second image processing model based on the training image data and the fourth image data. The second image processing model can be accurately adjusted to obtain the second image processing model until the training cutoff condition is met to obtain a trained second image processing model. The trained second image processing model can then process the image more accurately to achieve more accurate artifact removal.

[0109] In one embodiment, referring to Figure 4 and Figure 5 , Figure 4 is the image output by the second image processing model before training. Figure 4 There are artifacts in the image, and Figure 5 is the image output by the trained second image processing model. Figure 5 The image has no artifacts.

[0110] In one embodiment, an image processing method is also provided, which is applied to an electronic device, and the image processing method comprises the following steps:

[0111] Step A1: inputting original image data into a first image processing model to obtain first image data of a first image domain corresponding to the original image data.

[0112] Step A2, obtaining the processing operation required for the artifact removal process; determining the second image domain from at least two candidate image domains according to the processing operation; and the applicability of the artifact removal process of the second image domain is higher than the applicability of the artifact removal process of the first image domain.

[0113] Step A3, the first image domain is the RGB domain, and the second image domain is the SRGB domain or the YUV domain; if the second image domain is the SRGB domain, the first image data in the RGB domain is converted into the second image data in the SRGB domain; if the second image domain is the YUV domain, the first image data in the RGB domain is converted into the intermediate image data in the SRGB domain, and the intermediate image data in the SRGB domain is converted into the second image data in the YUV domain.

[0114] Step A4, obtaining the mask area of ​​each semantic object in the second image data; obtaining the de-artifacting type of the mask area for each semantic object; using the de-artifacting algorithm corresponding to the de-artifacting type to perform de-artifacting on the mask area to obtain the de-artifacting area; fusing each de-artifacting area to obtain the third image data of the second image domain.

[0115] Step A5: perform image domain inverse conversion on the third image data to obtain fourth image data in the first image domain.

[0116] Step A6, using the fourth image data and the original image data as a training data pair for artifact removal training; inputting the training data pair into the second image processing model, performing image processing on the original image data through the second image processing model to obtain training image data; adjusting parameters of the second image processing model according to the training image data and the fourth image data until a trained second image processing model is obtained when a training cutoff condition is met.

[0117] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0118] Based on the same inventive concept, the embodiment of the present application also provides an image processing device for implementing the above-mentioned image processing method. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above-mentioned method, so the specific limitations in one or more image processing device embodiments provided below can refer to the limitations on the image processing method above, and will not be repeated here.

[0119] In an exemplary embodiment, Figure 6 As shown, an image processing device is provided, including: a model output acquisition module 602, an image domain conversion module 604, a de-artifact processing module 606 and an image domain inverse conversion module 608, wherein:

[0120] The model output acquisition module 602 is used to input the original image data into the first image processing model to obtain the first image data of the first image domain corresponding to the original image data.

[0121] The image domain conversion module 604 is used to perform image domain conversion on the first image data to obtain second image data in a second image domain; the applicability of the de-artifacting processing in the second image domain is higher than the applicability of the de-artifacting processing in the first image domain.

[0122] The artifact removal processing module 606 is used to perform artifact removal processing on the second image data to obtain third image data in the second image domain.

[0123] The image domain inverse conversion module 608 is used to perform image domain inverse conversion on the third image data to obtain fourth image data in the first image domain.

[0124] The above-mentioned image processing device inputs the original image data into the first image processing model, obtains the first image data of the first image domain corresponding to the original image data, performs image domain conversion on the first image data, obtains the second image data of the second image domain, and the applicability of the de-artifacting processing in the second image domain is higher than the applicability of the de-artifacting processing in the first image domain. Then, the second image data in the second image domain is de-artifacted to obtain the third image data with effective de-artifacting, and then the third image data is inversely converted in the image domain to obtain the fourth image data of the first image domain with effective de-artifacting, and the fourth image data is more consistent with the real scene distribution, thereby improving the accuracy of image processing.

[0125] In one embodiment, the above-mentioned artifact removal processing module 606 is also used to obtain the mask area of ​​each semantic object in the second image data; for the mask area of ​​each semantic object, the mask area is subjected to artifact removal processing to obtain the artifact removal area; and each artifact removal area is fused to obtain the third image data of the second image domain.

[0126] In one embodiment, the above-mentioned artifact removal processing module 606 is also used to obtain the artifact removal type of the mask area for each semantic object; for the mask area of ​​each semantic object, the artifact removal algorithm corresponding to the artifact removal type is used to perform artifact removal processing on the mask area to obtain the artifact removal area.

[0127] In one embodiment, the apparatus further comprises a second image domain determining module, which is used to obtain processing operations required for artifact removal processing; and determine the second image domain from at least two candidate image domains according to the processing operations.

[0128] In one embodiment, the first image domain is an RGB domain, and the second image domain is an SRGB domain or a YUV domain; the above-mentioned image domain conversion module 604 is also used to convert the first image data in the RGB domain into the second image data in the SRGB domain if the second image domain is the SRGB domain; if the second image domain is the YUV domain, convert the first image data in the RGB domain into the intermediate image data in the SRGB domain, and convert the intermediate image data in the SRGB domain into the second image data in the YUV domain.

[0129] In one embodiment, the above-mentioned device also includes a training module, which is used to use the fourth image data and the original image data as a training data pair for artifact removal training; input the training data pair into the second image processing model, and perform image processing on the original image data through the second image processing model to obtain training image data; adjust the parameters of the second image processing model according to the training image data and the fourth image data until a trained second image processing model is obtained when a training cutoff condition is met.

[0130] Each module in the above-mentioned image processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in an electronic device in the form of hardware, or can be stored in a memory in an electronic device in the form of software, so that the processor can call and execute operations corresponding to each module above.

[0131] In an exemplary embodiment, an electronic device is provided. The electronic device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 7 As shown. The electronic device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the electronic device is used to exchange information between the processor and an external device. The communication interface of the electronic device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, an image processing method is implemented. The display unit of the electronic device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the electronic device casing, or an external keyboard, touchpad or mouse.

[0132] Those skilled in the art will understand that Figure 7The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0133] In one embodiment, an electronic device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0134] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0135] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0136] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0137] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0138] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0139] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An image processing method, characterized in that: The method comprises: Inputting the original image data into a first image processing model to obtain first image data of a first image domain corresponding to the original image data; Performing image domain conversion on the first image data to obtain second image data in a second image domain; the applicability of the artifact removal processing in the second image domain is higher than the applicability of the artifact removal processing in the first image domain; performing artifact removal processing on the second image data to obtain third image data in the second image domain; Perform image domain inverse conversion on the third image data to obtain fourth image data in the first image domain.

2. The method according to claim 1, characterized in that The performing artifact removal processing on the second image data to obtain third image data in the second image domain includes: Obtaining a mask region of each semantic object in the second image data; For each mask region of the semantic object, performing artifact removal processing on the mask region to obtain an artifact removal region; Each artifact removal area is fused to obtain third image data of the second image domain.

3. The method according to claim 2, characterized in that After obtaining the mask area of ​​each semantic object in the second image data, the method further includes: For each mask region of the semantic object, obtaining a de-artifacting type of the mask region; The step of performing artifact removal processing on the mask region of each semantic object to obtain the artifact removal region includes: For each mask region of the semantic object, a de-artifacting algorithm corresponding to the de-artifacting type is used to perform de-artifacting processing on the mask region to obtain a de-artifacting region.

4. The method according to claim 1, characterized in that: Before performing image domain conversion on the first image data to obtain second image data in a second image domain, the method further includes: Obtaining processing operations required for artifact removal; According to the processing operation, a second image domain is determined from at least two candidate image domains.

5. The method according to claim 4, characterized in that The first image domain is an RGB domain, and the second image domain is an SRGB domain or a YUV domain; performing image domain conversion on the first image data to obtain second image data in a second image domain includes: If the second image domain is an SRGB domain, converting the first image data in the RGB domain into second image data in the SRGB domain; If the second image domain is a YUV domain, the first image data in the RGB domain is image-domain-converted into intermediate image data in the SRGB domain, and the intermediate image data in the SRGB domain is image-domain-converted into second image data in the YUV domain.

6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: Using the fourth image data and the original image data as a training data pair for artifact removal training; Inputting the training data pair into a second image processing model, and performing image processing on the original image data by the second image processing model to obtain training image data; The parameters of the second image processing model are adjusted according to the training image data and the fourth image data until a trained second image processing model is obtained when a training cutoff condition is met.

7. An image processing device, characterized in that: The device comprises: A model output acquisition module, used for inputting the original image data into the first image processing model to obtain first image data of the first image domain corresponding to the original image data; An image domain conversion module, configured to perform image domain conversion on the first image data to obtain second image data in a second image domain; the applicability of the de-artifacting processing in the second image domain is higher than the applicability of the de-artifacting processing in the first image domain; an artifact removal processing module, configured to perform artifact removal processing on the second image data to obtain third image data in the second image domain; The image domain inverse conversion module is used to perform image domain inverse conversion on the third image data to obtain fourth image data in the first image domain.

8. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images