Image processing method, storage medium and terminal device

Abstract

The present invention discloses an image processing method, a storage medium and a terminal device, wherein the image processing method comprises obtaining a set of images to be processed, generating a denoised image corresponding to the set of images to be processed according to the set of images to be processed, and inputting the denoised image into a trained image processing model, and generating an output image corresponding to the denoised image through the image processing model. The present invention first obtains a plurality of images, and generates a denoised image according to the plurality of images, and then uses a trained image processing model obtained by deep learning based on the training image set to adjust the image color of the denoised image, so that the color quality and noise quality of the output image can be improved, thereby improving the image quality.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an image processing method, a storage medium and a terminal device. Background Art

[0002] Existing full-screen terminals generally include a display panel area and a camera area. The camera area is located on top of the display panel area. Although this can increase the screen-to-body ratio, the camera area will still occupy part of the display area, and it is not possible to truly achieve a full screen. Therefore, in order to achieve a full-screen terminal, it is necessary to install an imaging system under the display panel. Existing display panels generally include a substrate and a polarizer, etc. When light passes through the display panel, the display panel refracts the light, resulting in a low light transmittance, and on the other hand, the display panel absorbs the light, which will affect the quality of the captured image. For example, the color of the captured image does not match the shooting scene and the image noise increases. Summary of the invention

[0003] The technical problem to be solved by the present invention is to provide an image processing method, a storage medium and a terminal device in view of the deficiencies of the prior art.

[0004] In order to solve the above technical problems, the technical solution adopted by the present invention is as follows:

[0005] An image processing method, the method comprising:

[0006] Acquire a set of images to be processed, wherein the set of images to be processed includes a plurality of images;

[0007] According to the set of images to be processed, generating a denoised image corresponding to the set of images to be processed;

[0008] Inputting the denoised image into a first image processing model that has been trained, performing a decolorization process on the denoised image through the image processing model, and obtaining a processed image corresponding to the denoised image;

[0009] The processed image is input into a trained second image processing model, and the processed image is de-ghosted by the second image processing model to obtain an output image.

[0010] A computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in any of the above-mentioned image processing methods.

[0011] A terminal device comprises: a processor, a memory and a communication bus; the memory stores a computer-readable program executable by the processor;

[0012] The communication bus realizes the connection and communication between the processor and the memory;

[0013] When the processor executes the computer-readable program, the steps in any of the above-mentioned image processing methods are implemented.

[0014] Beneficial effects: Compared with the prior art, the present invention provides an image processing method, a storage medium and a terminal device, wherein the image processing method comprises obtaining a set of images to be processed, generating a denoised image corresponding to the set of images to be processed according to the set of images to be processed, and inputting the denoised image into a trained image processing model, and generating an output image corresponding to the denoised image through the image processing model. The present invention first obtains a plurality of images, and generates a denoised image according to the plurality of images, and then uses a trained image processing model obtained by deep learning based on the training image set to adjust the image color of the denoised image, so that the color quality and noise quality of the output image can be improved, thereby improving the image quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is a diagram of an application scenario of the image processing method provided by the present invention.

[0016] Figure 2 The present invention provides an image processing method according to an embodiment of the present invention.

[0017] Figure 3 This is a flowchart of step S20 in one embodiment of the image processing method provided by the present invention.

[0018] Figure 4 The present invention provides a flowchart of a process for acquiring adjacent image blocks in an embodiment of the image processing method provided by the present invention.

[0019] Figure 5 An example diagram of a designated area in an embodiment of the image processing method provided by the present invention.

[0020] Figure 6 The present invention provides a flowchart of a calculation process of a second weight parameter in an embodiment of the image processing method provided by the present invention.

[0021] Figure 7 A flowchart of the training process of the first image processing model in one embodiment of the image processing method provided by the present invention.

[0022] Figure 8 This is an example diagram of a first image in one embodiment of the image processing method provided in this embodiment.

[0023] Fig. 9This is an example diagram of a second image in one embodiment of the image processing method provided in this embodiment.

[0024] Fig.10 A flowchart of a process of determining an alignment method in one embodiment of the image processing method provided in this embodiment.

[0025] Fig.11 A schematic diagram of the principle of a first preset network model in one embodiment of the image processing method provided in this embodiment.

[0026] Fig.12 This is a workflow diagram of the first preset network model in one embodiment of the image processing method provided in this embodiment.

[0027] Fig.13 This is a flowchart of step M10 in one embodiment of the image processing method provided in this embodiment.

[0028] Fig.14 This is a flowchart of step M11 in one embodiment of the image processing method provided in this embodiment.

[0029] Fig.15 This is a flowchart of step M12 in one embodiment of the image processing method provided in this embodiment.

[0030] Fig.16 This is an example diagram of a denoised image in the image processing method provided in this embodiment.

[0031] Fig.17 This is an example diagram of a processed image corresponding to the denoised image in the image processing method provided in this embodiment.

[0032] Fig.18 A flowchart of the training process of the second image processing model in one embodiment of the image processing method provided by the present invention.

[0033] Fig.19 A schematic diagram of the structure of a second preset network model in an embodiment of the image processing method provided by the present invention.

[0034] Fig. 20 This is a flowchart of step L200 in one embodiment of the image processing method provided by the present invention.

[0035] Fig.21 This is an example diagram of a processed image in one embodiment of the image processing method provided by the present invention.

[0036] Fig. 22 An example diagram of an output image corresponding to a processed image in an embodiment of the image processing method provided by the present invention.

[0037] Fig.23 This is a structural principle diagram of the terminal device provided by the present invention. DETAILED DESCRIPTION

[0038] The present invention provides an image processing method, a storage medium and a terminal device. In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0039] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0040] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as herein.

[0041] After research, the inventor found that in order to realize the full screen of the terminal device, the front camera of the terminal device needs to be installed under the display panel. The existing display panel generally includes a substrate and a polarizer, and when light passes through the display panel, the display panel refracts the light, making the light transmittance low, and absorbs the light, which will affect the quality of the captured image. For example, the color of the captured image does not match the shooting scene, the image noise increases, and the image is blurred.

[0042] In order to solve the above problem, in an embodiment of the present invention, a set of images to be processed including multiple images is first obtained, and a denoised image is generated according to the set of images to be processed; then the trained first image processing model performs a decolorization process on the denoised image to obtain a processed image, and then the processed image is input into the trained second image processing model, and the second image processing model performs a deghosting process on the processed image. It can be seen that in an embodiment of the present invention, a denoised image is obtained by acquiring multiple images, and then the image color of the denoised image is adjusted by the first image processing model, and finally the second image processing model performs a deghosting process on the processed image after the decolorization process, which can improve the image quality.

[0043] For example, the embodiments of the present invention can be applied to Figure 1 The scenario shown. In this scenario, first, the terminal device 1 can collect a set of images to be processed, and input the set of images to be processed into the server 2, so that the server 2 obtains an output image based on the set of images to be processed, and then transmits the output image to the terminal device, so that the terminal device can obtain and display the output image. The server 2 can pre-store a trained first image processing model and a second image processing model, and respond to the set of images to be processed input by the terminal device 1, and generate a denoised image corresponding to the set of images to be processed according to the set of images to be processed; input the denoised image into the trained first image processing model, generate a processed image corresponding to the denoised image through the first image processing model, and finally generate an output image corresponding to the processed image through the second image model.

[0044] It is understandable that, in the above application scenario, although the actions of the embodiments of the present invention are described as being partially executed by the terminal device 1 and partially executed by the server 2, these actions may also be completely executed by the server 2, or completely executed by the terminal device 1. The present invention is not limited in terms of the execution subject, as long as the actions disclosed in the embodiments of the present invention are executed.

[0045] Furthermore, the image processing method can be used to process photos taken by a terminal device with an under-screen imaging system (such as an under-screen camera). For example, the photos taken by the terminal device with an under-screen imaging system (such as an under-screen camera) are used to obtain a set of images to be processed, and then a denoised image is generated according to the set of images to be processed, and then the denoised image is input as an input item into the trained first image processing model, and the photo is decolorized by the trained first image processing model to obtain a processed image, and finally an output image corresponding to the processed image is generated by the second image model, so that the taken picture can be quickly denoised, and the image color and ghosting can be adjusted to improve the image quality of the photos taken by the under-screen camera. Of course, in actual applications, the image processing method can be configured as an image processing function module in a terminal device with an under-screen imaging system (such as an under-screen camera). When the terminal device with the under-screen imaging system (such as an under-screen camera) takes a photo, the image processing function is started, and the photo taken by the terminal device through the image processing function is processed, so that the terminal device with the under-screen imaging system (such as an under-screen camera) can directly output high-quality photos.

[0046] It should be noted that the above application scenarios are only shown to facilitate understanding of the present invention, and the embodiments of the present invention are not limited in this respect. On the contrary, the embodiments of the present invention can be applied to any applicable scenarios.

[0047] The invention is further described below by describing embodiments in conjunction with the accompanying drawings.

[0048] This implementation provides an image processing method, such as Figure 2 As shown, the method includes:

[0049] S10: Obtain a set of images to be processed.

[0050] Specifically, the image set to be processed includes at least two images, and the acquisition method of each image in the image set to be processed may include: obtained by shooting through an imaging system (such as an under-screen camera, etc.), sent by an external device (such as a smart phone, etc.), and through a network (such as Baidu, etc.). In this embodiment, each image contained in the image set to be processed is a low-exposure image, and each denoised image in the image set to be processed is obtained by shooting through an imaging system (such as a camera, a camcorder, an under-screen camera, etc.), and each denoised image belongs to the same color space (such as RGB color space and YUV color space, etc.). For example, each denoised image is shot by an under-screen camera, and the base image and each adjacent image belong to the RGB color space.

[0051] Furthermore, the shooting scenes corresponding to the images to be processed in the set of images to be processed are the same, and the shooting parameters of the images to be processed may also be the same, wherein the shooting parameters may include ambient illumination and exposure parameters, wherein the exposure parameters may include aperture, door opening speed, sensitivity, focus, and white balance, etc. Of course, in practical applications, the shooting parameters may also include shooting angles and shooting ranges, etc.

[0052] Furthermore, since the noise level of the image captured by the imaging system is different under different ambient illuminations, for example, when the ambient illumination is low, the image captured by the imaging system carries more noise, and when the ambient illumination is high, the image captured by the imaging system carries less noise. In particular, for the under-screen imaging system, since the display panel has different absorption intensities for different light intensities, and the absorption degree of the display panel to light is nonlinear with the light intensity (for example, when the ambient illumination is low, the light intensity is low, and the proportion of light absorbed by the display panel is high; when the ambient illumination is high, the light intensity is high, and the proportion of light absorbed by the display panel is low), the noise intensity of the image A captured by the under-screen imaging system is higher than the noise intensity of the image B, wherein the ambient light intensity corresponding to the image A is less than the ambient light intensity corresponding to the image A. Therefore, for images with different noise intensities, different numbers of images can be used for synthesis, for example, the number of images required for images with high noise intensity is greater than the number of images required for images with low noise intensity. Accordingly, the number of images of the denoised images contained in the image set to be processed can be determined according to the shooting parameters corresponding to the image set to be processed, wherein the shooting parameters at least include the ambient illumination.

[0053] In addition, in order to determine the number of images to be processed according to the ambient illumination, the corresponding relationship between the ambient illumination interval and the number of images to be processed can be preset. After obtaining the ambient illumination, the ambient illumination interval in which the ambient illumination is located is first determined, and then the number of images to be processed corresponding to the ambient illumination interval is determined according to the corresponding relationship to obtain the number of images to be processed. For example, the correspondence between the ambient illumination interval and the number of images to be processed is: when the ambient illumination interval is [0.5, 1), the number of images to be processed corresponds to 8; when the ambient illumination is [1, 3), the number of images to be processed corresponds to 7; when the ambient illumination is [3, 10), the number of images to be processed corresponds to 6; when the ambient illumination interval is [10, 75), the number of images to be processed corresponds to 5; when the ambient illumination interval is [75, 300), the number of images to be processed corresponds to 4; when the ambient illumination interval is [300, 1000), the number of images to be processed corresponds to 3; when the ambient illumination is [1000, 5000), the number of images to be processed corresponds to 2.

[0054] Further, in an implementation of this embodiment, the image set to be processed is obtained by shooting through the under-screen imaging system, and the number of images to be processed contained in the image set to be processed is determined according to the ambient illumination when the under-screen imaging system shoots the images. The ambient illumination can be obtained when the under-screen imaging system is started, or can be obtained according to the first frame of image shot, or can be determined according to any one of the preset number of images shot by pre-shooting.

[0055] In one implementation of this embodiment, the ambient illumination is acquired when the under-screen imaging system is started. Accordingly, the acquisition process of the image set to be processed may be: when the under-screen imaging system is started, the ambient illumination is acquired, and the first number of images included in the image set to be processed is determined according to the acquired ambient illumination, and the first number of images are continuously acquired through the under-screen imaging system to obtain the image set to be processed. The first number of images can be determined according to the corresponding relationship between the preset ambient illumination and the number of images to be processed.

[0056] In one implementation of this embodiment, the ambient illumination is obtained based on the first frame of image captured. Accordingly, the process of obtaining the image set to be processed may be: first, the first frame of image is obtained through the under-screen imaging system, then the ISO value of the first frame of image is obtained, and the ambient illumination corresponding to the first frame of image is determined according to the ISO value, and finally, the second preset number of images included in the image set to be processed is determined according to the obtained ambient illumination, and the second number of images minus one image is continuously obtained through the under-screen imaging system to obtain the image set to be processed.

[0057] In one implementation of this embodiment, the ambient illumination is determined by taking a preset number of images in advance, and then determining the ambient illumination according to any one of the preset number of images taken. The acquisition process of the image set to be processed may be: firstly acquiring a preset number of images through the under-screen imaging system, and randomly selecting a third preset image from the acquired images, acquiring the ISO value of the third preset image, and determining the ambient illumination corresponding to the third preset image according to the ISO value, and finally determining the number of images (i.e., the third number of images) included in the image set to be processed according to the acquired ambient illumination. In addition, since the preset number of images has been acquired, the preset number can be compared with the third number of images. If the preset number is less than the third number of images, the fourth number of images is continuously acquired through the under-screen imaging system, wherein the fourth number of images is equal to the third number of images minus the preset number; if the preset number is equal to the third number of images, the acquisition operation of the image set to be processed is completed; if the preset number is greater than the third number of images, the third number of images is randomly selected from the acquired preset number of images to obtain the image set to be processed.

[0058] Further, in an implementation of this embodiment, when the preset number is greater than the third number of images, in order to make the image set to be processed include the third preset image, the third preset image can be added to the image set to be processed, and then the third number of images minus one image is selected from the acquired images. At the same time, in order to make the images in the image set to be processed continuous with the third preset image, the images included in the image set to be processed can be selected according to the order of taking pictures.

[0059] For example: assuming that the preset number is 5, the 5 images are recorded as image A, image B, image C, image D and image E in the order of shooting, the number of the third images is 3, and the third preset image is image C in the order of shooting time, then the images selected in the shooting order are image B and image D, respectively, so that the image set to be processed includes image B, image C and image D. Of course, in practical applications, it can be preferred to select from the third preset image in sequence in the shooting order, and when the number of images before the third preset image is insufficient, select from the third preset image in sequence in the shooting order; it can also be selected backward first, and when the number of backward images is insufficient, select forward; it can also be other selection methods, which are not specifically limited here, as long as the method can select images up to the fourth number of images.

[0060] S20: Generate a denoised image corresponding to the image set to be processed according to the image set to be processed.

[0061] Specifically, the image set to be processed includes a base image and at least one adjacent image, wherein the base image is the image reference of each image to be processed in the image set to be processed, and each adjacent image can be synthesized with the base image with the base image as the reference reference. Therefore, before generating a denoised image according to the image set to be processed, it is necessary to select an image in the image set to be processed as the base image, and use all images in the image set to be processed except the base image as adjacent images of the base image.

[0062] Furthermore, since the image set to be processed includes a basic image and at least one adjacent image, it is necessary to select a basic image from the acquired images. The basic image may be the first image in the acquisition order, or any image in the image set to be processed, or an image with the highest definition in the image set to be processed. In this embodiment, the basic image is the image with the highest definition in the processed image set, that is, the definition of the basic image is greater than or equal to the definition of any adjacent image.

[0063] Further, in an implementation of this embodiment, the basic image determination process may be: after all images included in the image set to be processed are obtained, the clarity of each image is obtained, and the obtained clarity is compared to select the image with the highest clarity, and the selected image is used as the basic image. The clarity of the image can be understood as the difference between the pixel value of the pixel point on the boundary of the object (or the boundary of the object) in the image and the pixel value of the pixel point adjacent to the boundary of the object (or the boundary of the object); it can be understood that if the difference between the pixel value of the pixel point on the boundary of the object (or the boundary of the object) in the image and the pixel value of the pixel point adjacent to the boundary of the object (or the boundary of the object) is larger, it means that the clarity of the image is higher, and conversely, if the difference between the pixel value of the pixel point on the boundary of the object (or the boundary of the object) in the image and the pixel value of the pixel point adjacent to the boundary of the object (or the boundary of the object) is smaller, it means that the clarity of the image is lower. That is to say, the clarity of the basic image is higher than the clarity of each adjacent image. It can be understood that, for each adjacent image, the difference between the pixel value of the pixel point on the boundary of the feature (or object boundary) in the basic image and the pixel value of the pixel point adjacent to the boundary of the feature (or object boundary) is greater than the difference between the pixel value of the pixel point on the boundary of the feature (or object boundary) in the adjacent image and the pixel value of the pixel point adjacent to the boundary of the feature (or object boundary).

[0064] For ease of understanding, an example is given below to illustrate that the clarity of the base image is higher than the clarity of the adjacent image. Assume that the image set to be processed includes image A and image B, and the image contents in image A and image B are exactly the same, wherein both image A and image B include pixel a and pixel b, pixel a is a pixel on the boundary of the feature (or object boundary) in the image, and pixel b is a pixel adjacent to the boundary of the feature (or object boundary); if the difference between the pixel value of pixel a and the pixel value of pixel b in image A is 10, and the difference between the pixel value of pixel a and the pixel value of pixel b in image B is 30, then it can be considered that the clarity of image B is higher than that of the training image A, therefore, image A can be used as the base image in the image set to be processed, and image B can be used as the adjacent image in the image set to be processed.

[0065] Further, in an implementation of this embodiment, when selecting a basic image from the image set to be processed according to clarity, there are multiple images with the same clarity (recorded as images C) in the image set to be processed, and the clarity of each image C is not less than the clarity of any image in the image set to be processed, then the multiple images C can all be used as basic images. At this time, an image C can be randomly selected from the multiple images C as the basic image, or the first image C can be selected from the multiple images C in the order of shooting as the basic image, or the last image C can be selected from the multiple images C in the order of shooting as the basic image.

[0066] Further, in one implementation of this embodiment, Figure 3 As shown, generating the denoised image corresponding to the image set to be processed according to the image set to be processed specifically includes:

[0067] S21, dividing the basic image into a plurality of basic image blocks, and respectively determining the adjacent image blocks corresponding to each basic image in each adjacent image;

[0068] S22, determining weight parameter sets corresponding to each basic image block; wherein the weight parameter sets corresponding to the basic image blocks include a first weight parameter and a second weight parameter, the first weight parameter is a weight parameter of the basic image block, and the second weight parameter is a weight parameter of an adjacent image block in an adjacent image corresponding to the basic image block;

[0069] S23, determining a denoised image according to the set of images to be processed and the weight parameter sets corresponding to the basic image blocks.

[0070] Specifically, in step S21, the basic image block is a partial image area of ​​the basic image, and the basic image blocks will form the basic image after being spliced. The basic image is divided into several basic image blocks, which means that the basic image is taken as a region, and the region is divided into several sub-regions, and the image region corresponding to each sub-region is a basic image block, wherein the region is divided into several sub-regions for dividing the region into several regions. For example, an 8*8 basic image can be divided into four 4*4 basic image blocks. Of course, in practical applications, the method of dividing the basic image into several basic image blocks in this embodiment can be flexibly selected according to the specific scene, as long as the method can be divided to obtain several basic image blocks. The adjacent image block is the image block corresponding to the basic image in the adjacent image, and the image block size of the adjacent image block is the same as the size of the basic image block corresponding to the adjacent image block, and the image content carried by the adjacent image block is the same as the image content carried by the basic image block. Determining the adjacent image blocks corresponding to the basic image block in each adjacent image refers to selecting an image block with the highest similarity to the basic image block within a specified area of ​​the adjacent image blocks, wherein the specified area is determined according to the area where the basic image block is located in the basic image.

[0071] Further, in one implementation of this embodiment, Figure 4 As shown, the step of respectively determining the adjacent image blocks corresponding to each basic image in each adjacent image specifically includes:

[0072] A10, determining a region range of the basic image block in the basic image, and determining a designated region in the adjacent image according to the region range;

[0073] A20. Select an adjacent image block in a specified area according to the basic image block, wherein the adjacent image block is an image block with the highest similarity to the basic image block in the specified area, and an image size of the adjacent image block is equal to an image size of the basic image block.

[0074] Specifically, the area range refers to a set of coordinate points formed by the pixel coordinates of the boundary pixel points of the area where the basic image block is located in the basic image. For example, the basic image block is a square area in the basic image, and the coordinate points of the four vertices of the basic image block are (10, 10), (10, 20), (20, 10) and (20, 20), respectively. Then the area range corresponding to the basic image block can be {(10, 10), (10, 20), (20, 10) and (20, 20)}.

[0075] The designated area is an image area in an adjacent image, and the area range corresponding to the basic image block may correspond to the area range corresponding to the designated area, that is, when the adjacent image is mapped to the basic image, the area range corresponding to the designated area corresponds to the area range corresponding to the basic image block. For example, in the basic image, the area range corresponding to the basic image block may be {(10,10), (10,20), (20,10) and (20,20)}, and in the adjacent image, the area range corresponding to the designated area may be {(10,10), (10,20), (20,10) and (20,20)}, then the area range corresponding to the basic image block may correspond to the area range corresponding to the designated area. In addition, the area range corresponding to the basic image block may also correspond to the area range corresponding to a sub-area of ​​the designated area, that is, when the adjacent image is mapped to the basic image, there is a sub-area in the area range corresponding to the designated area, and the area range of the sub-area corresponds to the area range corresponding to the basic image block. For example, the area range corresponding to the image area occupied by the basic image block in the basic image may be {(10,10), (10,20), (20,10) and (20,20)}, such as Figure 5 As shown, the area range corresponding to the image area 12 occupied by the specified area in the adjacent image can be {(9,9), (9,21), (21,9) and (21,21)}, then the specified area includes a sub-area 11, and the area range of the sub-area 11 is {(10,10), (10,20), (20,10) and (20,20)}, and the area range corresponding to the sub-area 11 corresponds to the area range corresponding to the basic image block.

[0076] Furthermore, in an implementation of the present embodiment, the area range corresponding to the basic image block may also correspond to the area range corresponding to the sub-area of ​​the designated area, and the designated area is obtained by translating each coordinate point in the set of coordinate points corresponding to the area range by a preset value along the horizontal or vertical axis away from the area range, wherein the area range is the area range corresponding to the basic image block. For example, the area range corresponding to the basic image block may be {(10,10), (10,20), (20,10) and (20,20)}, and the preset value is 5, then the area range of the designated area is {(5,5), (5,25), (25,5) and (25,25)}. In addition, the preset values ​​corresponding to different adjacent images may be different, and the preset values ​​corresponding to each adjacent image may be determined based on the displacement of the adjacent image relative to the basic image. Among them, the process of determining the preset value can be: for each adjacent image, respectively calculate the projections of the basic image and the adjacent image in the row and column directions, determine the displacement of the adjacent image relative to the basic image in the row and column according to the projection corresponding to the adjacent image and the projection corresponding to the basic image, and use the displacement as the preset value corresponding to the adjacent image, wherein the displacement can be calculated using the SAD algorithm.

[0077] Further, in step S22, the number of second weight parameters in the weight parameter set is the same as the number of adjacent images in the image set to be processed, and the second weight parameters in the weight parameter set correspond one-to-one to the adjacent images in the image set to be processed. Each adjacent image includes at least one adjacent image block corresponding to the basic image block, and each adjacent image block has a second weight parameter corresponding to the basic image block. Thus, the weight parameter set includes a first weight parameter and at least one second weight parameter, and each second weight parameter corresponds to an adjacent image block in an adjacent image corresponding to the basic image block. Among them, the first weight parameter can be pre-set to indicate the degree of similarity between the basic image block and itself; the second weight parameter is obtained based on the basic image block and its corresponding adjacent image.

[0078] Therefore, in a possible implementation of this embodiment, determining the weight parameter sets corresponding to the basic image blocks specifically includes:

[0079] For each basic image block, second weight parameters of adjacent image blocks corresponding to the basic image block are determined, and first weight parameters corresponding to the basic image block are acquired to obtain a weight parameter set corresponding to the basic image block.

[0080] Specifically, for each basic image block, there is at least one corresponding adjacent image block, wherein the number of adjacent image blocks corresponding to the basic image is equal to the number of adjacent images corresponding to the basic image. And for each adjacent image block corresponding to the basic image, the adjacent image block corresponds to a second weight parameter, so that the number of second weight parameters corresponding to the basic image block is equal to the number of adjacent images in the image set to be processed. In addition, the second weight parameter is calculated based on the similarity between the basic image block and the adjacent image block. Accordingly, in one implementation of this embodiment, Figure 6 As shown, the calculation of the second weight parameters of each adjacent image block corresponding to the basic image block specifically includes:

[0081] B10. For each adjacent image block, calculate a similarity value between the basic image block and the adjacent image block;

[0082] B20. Calculate a second weight parameter of the adjacent image block according to the similarity value.

[0083] Specifically, the similarity refers to the similarity between the basic image block and the adjacent image block, the adjacent image block is determined in the adjacent image according to the basic image block, and the image size of the adjacent image block is the same as the size of the basic image block, so that each pixel point included in the basic image block corresponds to each pixel point included in the adjacent image block, and for each pixel point in the basic image block, a pixel point corresponding to the pixel point can be found in the adjacent image block. Therefore, the similarity can be calculated based on the pixel value of each pixel point included in the basic image block and the pixel value of each pixel point included in the adjacent image block.

[0084] The specific process of calculating the similarity based on the pixel value of each pixel point included in the basic image block and the pixel value of each pixel point included in the adjacent image block can be: reading the first pixel value corresponding to each pixel point included in the basic image block and the second pixel value corresponding to each pixel point included in the adjacent image block; for each first pixel value, calculating the difference between the first pixel value and the second pixel value corresponding to it; and then calculating the similarity value of the basic image block and the adjacent image block based on all the calculated differences, wherein the similarity value can be the average of the absolute values ​​of the calculated differences, for example, the first pixel value A and the second pixel value A are calculated. The difference between the first pixel value A and the second pixel value B, then, the similarity between the basic image block and the adjacent image block can be determined according to the difference between the first pixel value A and the second pixel value A, and the difference between the first pixel value B and the second pixel value B. Specifically, the similarity value can be the average of the absolute value of the difference between the first pixel value A and the second pixel value A and the absolute value of the difference between the first pixel value B and the second pixel value B. Therefore, the larger the similarity value is, the lower the similarity between the basic image block and the adjacent image block is. Conversely, the smaller the similarity value is, the higher the similarity between the basic image block and the adjacent image block is.

[0085] At the same time, in this embodiment, for each basic image block of the basic image and its corresponding adjacent image block and The similarity d i The calculation formula can be:

[0086]

[0087] Wherein, j is the pixel index, j=1, 2, ..., M, M is the number of pixels contained in the basic image block (the number of pixels contained in the adjacent image block is the same as that in the basic image block), is the pixel value of the jth pixel in the basic image block; is the pixel value of the pixel in the adjacent image block corresponding to the j-th pixel in the basic image block, i represents the i-th basic image block, i=1, 2, ..., N, where N is the number of basic image blocks.

[0088] Further, according to the calculation formula of the similarity value, it can be known that the similarity value is related to the image noise intensity of the image in the image set to be processed, and the difference between the image content of the basic image block and the image content of the adjacent image block. Specifically, when the image noise intensity is high or the difference between the image content of the basic image block and the image content of the adjacent image block is large, the similarity value is large; conversely, when the image noise intensity is low and the difference between the image content of the basic image block and the image content of the adjacent image block is small, the similarity value is small. Then, using the adjacent image block with a large similarity value for subsequent synthesis operation will result in poor synthesis effect, so that after obtaining the similarity value of the basic image block and each adjacent image block, a second weight parameter can be configured for each adjacent image block according to the similarity value corresponding to each adjacent image block. The second weight parameter is negatively correlated with the similarity value, that is, the larger the similarity value, the smaller the second weight parameter; conversely, the smaller the similarity value, the larger the second weight parameter. In this way, by assigning a lower weight value to adjacent images with low similarity, distortion problems such as smearing after fusion are prevented.

[0089] Exemplarily, in an implementation of this embodiment, calculating the second weight parameter of the adjacent image block according to the similarity value specifically includes:

[0090] C10. When the similarity value is less than or equal to the first threshold, using the first preset parameter as the second weight parameter of the adjacent image block;

[0091] C20, when the similarity value is greater than a first threshold value and less than or equal to a second threshold value, calculating a second weight parameter of the adjacent image block according to the similarity value, the first threshold value and the second threshold value;

[0092] C30. When the similarity value is greater than a second threshold, a second preset parameter is used as a second weight parameter of the adjacent image block.

[0093] It should be noted that, in this embodiment, B20 may include only any one step, any two steps, or all steps among C10, C20, and C30, that is, in this embodiment, B20 may include C10 and / or C20 and / or C30.

[0094] Specifically, the first threshold and the second threshold are both used to measure the similarity between the basic image block and the adjacent image block, and the second threshold is greater than the first threshold. Then, when the similarity value is less than the first threshold, it can be known from the relationship between the similarity value and the similarity that the similarity between the basic image block and the adjacent image block is high, so the second weight parameter value corresponding to the adjacent image block is large, and when the similarity value is greater than the second threshold, it can be known from the relationship between the similarity value and the similarity that the similarity between the basic image block and the adjacent image block is low, so the second weight parameter value corresponding to the adjacent image block is small. It can be seen that the first preset parameter is greater than the second preset parameter, and the third parameter calculated by calculating the adjacent image block according to the similarity value, the first threshold and the second threshold is between the first preset parameter and the second preset parameter.

[0095] Further, in an implementation of the embodiment, the calculation process of the third parameter may be: firstly calculate the first difference between the similarity value and the second threshold, then calculate the second difference between the first threshold and the second threshold, and then calculate the ratio of the first difference and the second difference, and use the ratio as the second weight parameter of the adjacent image block. In addition, it can be obtained from the calculation process of the third parameter that the value range of the third parameter is 0-1, and the first preset parameter is greater than the third parameter, and the second preset parameter is less than the third parameter, so that the first preset parameter can be set to 1 and the second preset parameter can be set to 0. Therefore, the corresponding relationship between the second weight parameter and the similarity value can be expressed as:

[0096]

[0097] Among them, w i is the second weight parameter, t1 is the first threshold, t2 is the second threshold, d i The similarity value i represents the i-th basic image block, i=1, 2, ..., N, where N is the number of basic image blocks.

[0098] Of course, it is worth explaining that the similarity is positively correlated with the weight coefficient, that is, the higher the similarity between the base image and the adjacent image, the larger the weight coefficient corresponding to the adjacent image block, and vice versa, the lower the similarity between the base image and the adjacent image, the lower the weight coefficient corresponding to the adjacent image block. As for the base image block, the comparison object for the base image block to determine the similarity is the base image block itself, so the similarity between the base image block and itself is greater than or equal to the similarity between the adjacent image block and the base image block, and accordingly, the first weight parameter is greater than or equal to the second weight coefficient. At the same time, since it can be seen from the calculation formula of the second weight parameter that the maximum value of the second weight coefficient is 1, then in one implementation of this embodiment, the first weight coefficient corresponding to the base image block can be equal to the maximum value of the second weight parameter, that is, the first weight parameter is 1.

[0099] Further, the first threshold and the second threshold may be pre-set, or may be determined according to the similarity value of the adjacent image block corresponding to the basic image block. In the present embodiment, the first threshold and the second threshold are determined according to the similarity value of each adjacent image block corresponding to the basic image block. The process of determining the first threshold and the second threshold may be: respectively obtaining the similarity value of each adjacent image block, respectively calculating the mean and the standard deviation of each similarity value, and then calculating the first threshold and the second threshold according to the mean and the standard deviation, so that the first threshold and the second threshold are determined by the similarity value of each adjacent image block, so that the first threshold and the second threshold can be adaptively adjusted according to the similarity value of each adjacent image, so that the first threshold and the second threshold can be adaptively adjusted according to the noise intensity of each adjacent image, avoiding the poor image denoising effect caused by the first threshold and the second threshold being too large, and the image blurring caused by the first threshold and the second threshold being too small, thereby improving the clarity of the image on the basis of ensuring the image denoising effect.

[0100] Further, in an implementation of this embodiment, the calculation formulas of the first threshold t1 and the second threshold t2 are respectively:

[0101] t1=μ+s min ×σ

[0102] t2=μ+s max ×σ

[0103]

[0104]

[0105] Among them, S min and S max is a constant, d max is a constant, L represents d i <d max The number of adjacent image blocks, i = 1, 2, ..., L.

[0106] In addition, the image noise intensity of the image in the image set to be processed and the accuracy of the selection of adjacent image blocks affect the similarity value. The accuracy of the selection of adjacent image blocks can cause a large change in the similarity value, so that when the similarity value between the adjacent image block and the basic image block is greater than the preset value d max When d is , the default is that the image content of the basic image block is too different from the image content of the adjacent image block, and the adjacent image block is regarded as an invalid adjacent image block (that is, the invalid adjacent image block is discarded and is not used as an adjacent image block of the basic image block). i ≥d maxIf there is an adjacent image block with a large image content, it can be considered that the image content of the basic image block is too different from that of the adjacent image block, so there is no need to determine the first threshold and the second threshold corresponding to the adjacent image block, which improves the calculation speed of the weight parameter set corresponding to the basic image block. At the same time, the adjacent image block with a large image content difference from the basic image block can avoid the problem of smearing caused by the adjacent image blocks with a large image content difference during image fusion, which causes the output image distortion.

[0107] Further, in step S23, the output image is composed of a plurality of output image blocks, and the output image block is calculated based on the basic image block, the adjacent image blocks corresponding to the basic image block, and the weight parameter set corresponding to the basic image block. For example, for each pixel in the basic image block, the first pixel value of the pixel and the second pixel value of the pixel corresponding to the pixel in each adjacent image block are obtained, and then the second weight parameter corresponding to each adjacent image block and the first weight parameter corresponding to the basic image are used as weighting coefficients, and the first pixel value and each second pixel value are weighted to obtain the pixel value of each pixel in the output image block. Thus, when determining the output image based on the image set to be processed and the weight parameter set corresponding to each basic image block, for each basic image block, the basic image block and its corresponding adjacent image blocks can be weighted to obtain the output image block corresponding to the basic image block, wherein, in the weighting process of the basic image block and each adjacent image block, the weighting coefficient of the basic image block is the first weight system in the weight parameter set, and the weighting coefficient of each adjacent image block is the second weight parameter corresponding to each adjacent image block in the weight parameter set. In addition, after each output image block is calculated, the output image is generated according to the calculated output image blocks, wherein the output image is generated according to the output image blocks by replacing the corresponding basic image blocks in the basic image with each output image block, or the output image is obtained by splicing the output image blocks.

[0108] For example: assuming that the image set to be processed includes a basic image and four adjacent images, then for each basic image block, the basic image block corresponds to four adjacent image blocks, which are respectively recorded as the first adjacent image block, the second adjacent image block, the third adjacent image block and the fourth adjacent image block, and are arranged in the order of shooting: basic image block, first adjacent image block, second adjacent image block, third adjacent image block and fourth adjacent image block; then when determining the output image block corresponding to the basic image block, for each pixel point in the basic image block, the pixel value of the pixel point is A, and the pixel value of the pixel point corresponding to the pixel point in the first adjacent image block is B. , the pixel value of the pixel corresponding to the pixel in the second adjacent image block is C, the pixel value of the pixel corresponding to the pixel in the third adjacent image block is D, the pixel value of the pixel corresponding to the pixel in the fourth adjacent image block is E, and the first weight parameter corresponding to the basic image block is a, the second weight parameter corresponding to the first adjacent image block is b, the second weight parameter corresponding to the second adjacent image block is c, the second weight parameter corresponding to the third adjacent image block is d, and the second weight parameter corresponding to the fourth adjacent image block is e, then the pixel value of the output pixel corresponding to the pixel = (A*a+B*b+C*c+D*d+E*e) / 5.

[0109] S30, inputting the denoised image into a first image processing model that has been trained, performing a decolorization process on the denoised image through the image processing model, and obtaining a processed image corresponding to the denoised image.

[0110] Specifically, the denoised image is generated based on the image set to be processed, and the first image processing model may be pre-trained by an image device (e.g., a mobile phone with an under-screen camera) that processes the denoised image, or may be trained by other devices and then the file corresponding to the first image processing model is transplanted to the image device. In addition, the image device may use the first image processing model as an image processing function module. When the image device obtains the denoised image, the image processing function module is started and the denoised image is input into the first image processing model.

[0111] Furthermore, the first image processing model is trained based on a training image set, such as Figure 7 As shown, the training process of the first image processing model may be:

[0112] M10. The first preset network model generates a generated image corresponding to a first image in a training image set according to the first image in the training image set, wherein the training image set includes a plurality of training image groups, each training image group includes a first image and a second image, and the first image is a color cast image corresponding to the second image;

[0113] M20. The first preset network model modifies model parameters according to the second image corresponding to the first image and the generated image corresponding to the first image, and continues to execute the step of generating a generated image corresponding to the first image according to the first image in the next training image group in the training image set, until the training status of the first preset network model meets the preset conditions to obtain the image processing model.

[0114] Specifically, in the step M10, the first preset network model may adopt a deep network learning model, and the training image set includes a plurality of training image groups with different image contents, each training image group includes a first image and a second image, and the first image is a color cast image corresponding to the second image. Wherein, the first image being a color cast image corresponding to the second image means that the first image corresponds to the second image, the first image and the second image present the same image scene, and the number of first target pixel points in the first image that meet the preset color cast condition meets the preset number condition. It can be understood that the second image is a normally displayed image, and there are a number of first target pixel points in the first image that meet the preset color cast condition, and the number of the first target pixel points meets the preset condition. For example, the second image is as follows Fig. 9 The first image is as shown in Figure 8 The images shown in the figure, wherein the image content of the first image is the same as the image content of the second image, but the color of the apple in the first image is different from the color of the apple in the second image. For example, Figure 8 In the first image, the color of the apple in the first image is green and blue; Fig. 9 In the second image, the color of the apple in the second image is dark green.

[0115] Further, the preset color cast condition is that the error between the display parameter of the first target pixel in the first image and the display parameter of the second target pixel in the second image satisfies the preset error condition, and there is a one-to-one correspondence between the first target pixel and the second target pixel. Wherein, the display parameter is a parameter used to reflect the color corresponding to the pixel, for example, the display parameter can be the RGB value of the pixel, wherein the R value is the red channel value, the G value is the green channel value, and the B value is the blue channel value; it can also be the hsl value of the pixel, wherein the h value is the hue value, the l is the brightness value, and the s is the saturation value. In addition, when the display parameter is the RGB value of the pixel, the display parameter of any pixel in the first image and the second image includes three display parameters: R value, G value and B value; when the display is the hls value of the pixel, the display parameter of any pixel in the first image and the second image includes three display parameters: h value, l value and s value.

[0116] The preset error condition is used to measure whether the first target pixel is a pixel that meets the preset color cast condition, wherein the preset error condition is a preset error threshold, and the error meets the preset error condition when the error is greater than or equal to the preset error threshold. In addition, the display parameter includes several display parameters, for example, the display parameter is the RGB value of the pixel, and the display parameter includes three display parameters: R value, G value, and B value. When the display parameter is the hsl value of the pixel, the display parameter includes three display parameters: h value, l value, and s value. Thus, the error can be the maximum value of the error of each display parameter in the display parameter, or the minimum value of the error of each display parameter in the display parameter, or the average value of the error of all display parameters. For example, here the display parameter is the RGB value of the pixel point for explanation, the display parameter of the first target pixel point is (55, 86, 108), and the display parameter of the second target pixel point is (58, 95, 120), then the error values ​​of each display parameter are 3, 9 and 12 respectively; therefore, when the error between the first target pixel point and the second target pixel point is the maximum error of each display parameter, the error is 12; when the error between the first target pixel point and the second target pixel point is the minimum error of each display parameter, the error is 3; when the error between the first target pixel point and the second target pixel point is the average error of all display parameters, the error is 8; it should be noted that, in a possible implementation method, only one parameter (such as R, G or B) or the error of any two parameters in RGB can be referred to, and the same applies when the display parameter is the hsl value of the pixel point.

[0117] Furthermore, there is a one-to-one correspondence between the second target pixel point used to calculate the error with the first target pixel point and the first target display point. It can be understood that for the first target pixel point, there is a unique second target pixel point in the second image that corresponds to the first target pixel point, wherein the correspondence between the first target pixel point and the second target pixel point refers to the pixel position of the first target pixel point in the first image corresponding to the pixel position of the second target pixel point in the second image. For example, the pixel position of the first target pixel point in the first image is (5,6), and the pixel position of the second target pixel point in the second image is (5,6). In addition, the first target pixel point can be any pixel point in the first image, or any pixel point in the target area in the first image, wherein the target area can be the area where the object is located in the first image, wherein the area where the object is located can be the area corresponding to the person or object in the image. For example, as Figure 8As shown, the target area is the area where the apple is located in the first image. That is to say, all pixels in the first image may be color-cast compared with the second image, that is, all pixels in the first image are first target pixels, or only a part of the pixels may be color-cast compared with the second image, that is, some pixels in the first image are first target pixels. For example, when only pixels in a part of an area (such as the area corresponding to the apple in the figure) in an image are color-cast compared with the second image, the image can also be understood as a color-cast image corresponding to the second image, that is, the first image.

[0118] Further, the first image and the second image correspond to each other in that the image size of the first image is equal to the image size of the second image, and the first image and the second image correspond to the same image scene. The first image and the second image correspond to the same image scene in that the similarity between the image content carried by the first image and the image content carried by the second image reaches a preset threshold, and the image size of the first image is the same as the image size of the second image, so that when the first image and the second image overlap, the coverage rate of the object carried by the first image to the object corresponding to it in the second image reaches a preset condition. Among them, the preset threshold value can be 99%, and the preset condition can be 99.5%, etc. In actual applications, the first image can be obtained by taking pictures through an under-screen imaging system; the second image can be obtained by taking pictures through a normal on-screen imaging system (such as an on-screen camera), or it can be obtained through the network (such as Baidu), or it can be sent through other external devices (such as smart phones).

[0119] In a possible implementation of this embodiment, the second image is captured by a normal on-screen imaging system, and the second image and the first image have the same shooting parameters. The shooting parameters may include exposure parameters of the imaging system, and the exposure parameters may include aperture, shutter speed, sensitivity, focus, and white balance. Of course, in actual applications, the shooting parameters may also include ambient light, shooting angle, and shooting range. For example, the first image is as follows: Figure 8 The image shown is obtained by shooting a scene with the under-screen camera, and the second image is as shown Fig. 9 The image shown is obtained by photographing the scene through the on-screen camera.

[0120] Further, in an implementation of this embodiment, in order to reduce the impact of the image difference between the first image and the second image on the training of the first preset network model, the image content of the first image and the image content of the second image may be exactly the same. That is, the first image and the second image have the same image content, which means that the object content of the first image is the same as the object content of the second image, the image size of the first image is the same as the image size of the second image, and when the first image and the second image overlap, the object in the first image can cover the corresponding object in the second image.

[0121] For example: the image size of the first image is 400*400, the image content of the first image is a circle, and the center of the circle in the first image is located at (200, 200) in the first image, and the radius is 50 pixels. Then, the image size of the second image is 400*400, the image content of the second image is also a circle, the center of the circle in the second image is located at (200, 200) in the second image, and the radius is 50 pixels; when the first image is placed on the second image and overlaps with the second image, the first image covers the second image, and the circle in the first image overlaps with the circle in the second image.

[0122] Further, when the second image is obtained by taking a normal on-screen imaging system, since the first image and the second image are taken by two different imaging systems, when the imaging system is replaced, the shooting angle and / or shooting position of the on-screen imaging system and the under-screen imaging system may change, so that the first image and the second image are not aligned in space. Therefore, in a possible implementation of this embodiment, when the second image is taken by the on-screen imaging system and the first image is taken by the under-screen imaging system, the on-screen imaging system and the under-screen imaging system can be set on the same fixed frame, and the on-screen imaging system and the under-screen imaging system can be arranged side by side on the fixed frame, and the on-screen imaging system and the under-screen imaging system are kept in contact. At the same time, the on-screen imaging system and the under-screen imaging system are respectively connected to a wireless device (such as a Bluetooth watch, etc.), and the shutters of the on-screen imaging system and the under-screen imaging system are triggered by the wireless device, so that the position change of the on-screen imaging system and the under-screen imaging system during the shooting process can be reduced, and the alignment of the first image and the second image in space can be improved. Of course, the shooting time and shooting range of the on-screen imaging system and the under-screen imaging system are the same.

[0123] In addition, although during the shooting process of the first image and the second image, the shooting position, shooting angle, shooting time, exposure factor, etc. of the under-screen imaging system and the on-screen imaging system can be fixed. However, due to environmental parameters (such as light intensity, wind blowing the imaging system, etc.), the first image captured by the under-screen imaging system and the second image captured by the on-screen imaging system may still be spatially misaligned. Therefore, before the first image in the training image set is input into the first preset network model, the first image and the second image in each training image group in the training image set may be aligned, so that in one implementation of this embodiment, the first preset network model generates a generated image corresponding to the first image according to the first image in the training image set, and also includes:

[0124] N100. For each training image group in the training image set, align the first image in the training image group with the second image corresponding to the first image to obtain an aligned image aligned with the second image, and use the aligned image as the first image.

[0125] Specifically, the alignment process for each group of training image groups in the training image set refers to performing alignment processing on each group of training image groups in the training image set. The alignment processing may be that after acquiring the training image set, each group of training image groups is aligned to obtain an aligned training image group, and after all groups of training image groups are aligned, the step of inputting the first image in each group of training image groups into the first preset network model is performed; of course, it may also be that before the first image in each group of training image groups is input into the first preset network model, the group of training image groups is aligned to obtain the aligned training image group corresponding to the group of training images, and then the first image in the aligned training image group is input into the first preset network model. In this embodiment, the alignment processing is performed on each group of training image groups after acquiring the training image set, and after all training image groups have completed the alignment processing, the operation of inputting the first image in the training image set into the first preset network model is performed.

[0126] Further, in the process of processing the first image and the second image correspondingly, the first image is used as a reference image and the second image is used as a benchmark image, wherein the alignment process is based on the benchmark image. It can be understood that the alignment process of the reference image in the training image group with the benchmark image corresponding to the reference image is the alignment process of the reference image in the training image group with the benchmark image corresponding to the reference image, and the alignment process of the reference image in the training image group with the benchmark image corresponding to the reference image can be based on the base image, aligning the pixels in the reference image with the pixels corresponding to the base image, so that the alignment rate of the pixels in the reference image and the pixels in the benchmark image can reach a preset value, for example, 99%. Wherein, the alignment of the pixels in the reference image with the pixels corresponding to the benchmark image means that: for the reference pixel in the reference image and the benchmark pixel in the benchmark image corresponding to the reference pixel, if the pixel coordinates corresponding to the reference pixel are the same as the pixel coordinates corresponding to the benchmark pixel, then the reference pixel is aligned with the benchmark pixel; if the pixel coordinates corresponding to the reference pixel are different from the pixel coordinates corresponding to the benchmark pixel, then the reference pixel is aligned with the benchmark pixel. The alignment image refers to an image obtained by aligning the reference image, and each pixel in the alignment image has the same pixel coordinates as its corresponding pixel in the reference image. In addition, after obtaining the alignment image, the alignment image is used to replace its corresponding reference image to update the training image group, so that the reference image and the reference image in the updated training image group are spatially aligned.

[0127] Furthermore, since the alignment degrees of the reference images and the benchmark images in different training image groups are different, different alignment methods can be used for the reference images and the benchmark images with different alignment degrees on the basis of achieving the alignment, so that each training image group can be aligned using an alignment method with low complexity. Fig.10 As shown, the alignment process of the reference image in the training image group with the benchmark image corresponding to the reference image specifically includes:

[0128] N11, obtaining a pixel deviation between a reference image in the training image group and a benchmark image corresponding to the reference image;

[0129] N12. Determine an alignment method corresponding to the reference image according to the pixel deviation, and use the alignment method to align the reference image with the benchmark image.

[0130] Specifically, the pixel deviation refers to the total number of reference pixels in the reference image that are not aligned with the reference pixel in the reference image that corresponds to the reference pixel. The pixel deviation can be obtained by obtaining the reference coordinates of each reference pixel in the reference image and the reference coordinates of each reference pixel in the reference image, and then comparing the reference coordinates of the reference pixel with the reference coordinates of the reference pixel corresponding to it. If the reference coordinates are the same as the reference coordinates, it is determined that the reference pixel is aligned with the reference pixel corresponding to it; if the reference coordinates are not the same as the reference coordinates, it is determined that the reference pixel is not aligned with the reference pixel corresponding to it, and finally the total number of all the misaligned reference pixels is obtained to obtain the pixel deviation. For example, when the reference coordinates of the reference pixel point in the reference image are (200, 200) and the reference coordinates of the reference pixel point corresponding to the reference pixel point in the reference image are (201, 200), the reference pixel point is not aligned with the reference pixel point, and the total number of non-aligned reference pixels is increased by one; when the reference coordinates of the reference pixel point in the reference image are (200, 200) and the reference coordinates of the reference pixel point corresponding to the reference pixel point in the reference image are (200, 200), the reference pixel point is aligned with the reference pixel point, and the total number of non-aligned reference pixels remains unchanged.

[0131] Furthermore, in order to determine the correspondence between the pixel deviation and the alignment method, a deviation threshold may need to be set. When the pixel deviation of the reference image is obtained, the obtained pixel deviation may be compared with the preset deviation threshold to determine the alignment method corresponding to the pixel deviation. Therefore, in an implementation of this embodiment, determining the alignment method corresponding to the reference image according to the pixel deviation, and aligning the reference image with the benchmark image using the alignment method specifically includes:

[0132] N121. When the pixel deviation is less than or equal to a preset deviation threshold, aligning the reference image with the reference image as a reference according to the mutual information between the reference image and the reference image;

[0133] N122. When the pixel deviation is greater than the preset deviation threshold, extract the reference pixel point set of the reference image and the benchmark pixel point set of the benchmark image, wherein the reference pixel point set includes several reference pixel points in the reference image, and the benchmark pixel point set includes several benchmark pixel points in the benchmark image, and the benchmark pixel points in the benchmark pixel point set correspond one-to-one to the reference pixel points in the reference pixel point set; for each reference pixel point in the reference pixel point set, calculate the coordinate difference between the reference pixel point and the benchmark pixel point corresponding to it, and adjust the position of the reference pixel point according to the coordinate difference corresponding to the reference pixel point, so as to align the reference pixel point with the benchmark pixel point corresponding to the reference pixel point.

[0134] Specifically, the preset deviation threshold is pre-set, for example, the preset deviation threshold is 20. When the pixel deviation is less than or equal to the preset deviation threshold means that when the pixel deviation is compared with the preset deviation threshold, the pixel deviation is less than or equal to the preset deviation threshold. When the pixel deviation is less than or equal to the preset deviation threshold, it means that the spatial deviation between the reference image and the benchmark image is small. At this time, the reference image and the benchmark image can be aligned according to the mutual information between the reference image and the benchmark image. In this embodiment, the process of aligning the reference image and the benchmark image with the mutual information between the reference image and its corresponding benchmark image can adopt an image registration method, in which the mutual information is used as a metric criterion, and the metric criterion is iteratively optimized by an optimizer to obtain an alignment parameter, and the reference image and the benchmark image are aligned by the aligner that aligns the alignment parameter, which ensures the basis of the alignment effect of the reference image and the benchmark image, reduces the complexity of the alignment of the reference image and the benchmark image, and thus improves the alignment efficiency. In this embodiment, the optimizer mainly adopts translation and rotation transformation to optimize the metric criterion through the translation and rotation transformation.

[0135] Further, if the pixel deviation is greater than the preset deviation threshold, it means that the reference image and the benchmark image are highly misaligned in space, and the alignment effect needs to be considered. Therefore, the reference image and the benchmark image can be aligned by selecting a reference pixel set in the reference image and a benchmark pixel set in the benchmark image. The reference pixels of the reference pixel set correspond to the benchmark pixels in the benchmark pixel set one by one, so that for any reference pixel in the reference pixel set, a benchmark pixel can be found in the benchmark pixel set, and the position of the benchmark pixel in the benchmark image corresponds to the position of the reference pixel in the reference image. In addition, the reference pixel set and the benchmark pixel set can be determined according to the corresponding relationship between the reference pixel and the benchmark pixel after the reference pixel set / the benchmark pixel set is obtained. For example, the reference pixel set is generated by randomly selecting a plurality of reference pixels in the reference image, and the benchmark pixel is determined according to each reference pixel contained in the reference pixel set.

[0136] At the same time, in this embodiment, the reference pixel set and the reference pixel set are both obtained by scale-invariant feature transform (SIFT), that is, the reference pixel in the reference pixel set is the first SIFT feature point in the reference image, and the reference pixel in the reference pixel set is the second SIFT feature point in the reference image. Correspondingly, the calculation of the coordinate difference between the reference pixel and the corresponding reference pixel is to match the first SIFT feature point in the reference pixel with the second SIFT feature point in the reference pixel set point-to-point to obtain the coordinate difference between each first SIFT feature point and each corresponding second SIFT feature point, and transform the position of the first SIFT feature point according to the coordinate difference corresponding to the first SIFT feature point, so as to align the reference pixel with the second SIFT feature point corresponding to the first SIFT feature point, so that the first SIFT feature point in the reference image is in the same position as the second SIFT feature point in the reference image, thereby realizing the alignment of the reference image with the reference image.

[0137] Further, in one implementation of this embodiment, Fig.11 , 12 and Fig.13 As shown, the first preset network model includes a downsampling module 100 and a transformation module 200. Accordingly, the first preset network model generates a generated image corresponding to the first image according to the first image in the training image set, which may specifically include:

[0138] M11. Inputting a first image in the training image set into the downsampling module, and obtaining a bilateral grid corresponding to the first image and a guidance image corresponding to the first image through the downsampling module, wherein a resolution of the guidance image is the same as a resolution of the first image;

[0139] M12. Input the guiding image, the bilateral grid and the first image into the transformation module, and generate a generated image corresponding to the first image through the transformation module.

[0140] Specifically, the bilateral grid 10 is a three-dimensional bilateral grid obtained by adding a one-dimensional dimension representing pixel intensity to the pixel coordinates of the two-dimensional image, wherein the three dimensions of the three-dimensional bilateral grid are the horizontal axis and the vertical axis in the pixel coordinates of the two-dimensional image, and the added dimension representing pixel intensity. The guidance image is obtained by performing pixel-level operations on the first image, and the resolution of the guidance image 50 is the same as that of the first image. For example, the guidance image 50 is a grayscale image corresponding to the first image.

[0141] Further, since the downsampling module 100 is used to output the bilateral grid 10 and the guidance image 50 corresponding to the first image, the downsampling module 100 includes a downsampling unit 70 and a convolution unit 30, the downsampling unit 70 is used to output the bilateral grid 10 corresponding to the first image, and the convolution unit 30 is used to output the guidance image 50 corresponding to the first image. Fig.11 , 12 and Fig.14 As shown, the step of inputting the first image in the training image set into the downsampling module, and obtaining the bilateral grid parameters corresponding to the first image and the guidance image corresponding to the first image through the downsampling module specifically includes:

[0142] M111, inputting the first image in the training image set into the downsampling unit and the convolution unit respectively;

[0143] M112. Obtain a bilateral grid corresponding to the first image through the downsampling unit, and obtain a guidance image corresponding to the first image through the convolution unit.

[0144] Specifically, the downsampling unit 70 is used to downsample the first image to obtain a feature image corresponding to the first image, and generate a bilateral grid corresponding to the first image based on the feature image, wherein the number of spatial channels of the feature image is greater than the number of spatial channels of the first image. The bilateral grid is generated based on local features and global features of the feature image, wherein the local features are features extracted from a local area of ​​the image, such as edges, corners, lines, curves, and attribute areas, etc. In this embodiment, the local features may be regional color features. The global features refer to features that represent the attributes of the entire image, such as color features, texture features, and shape features. In this embodiment, the global features may be color features of the entire image.

[0145] Further, in a possible implementation of the present embodiment, the downsampling unit 70 includes a downsampling layer, a local feature extraction layer, a global feature extraction layer and a fully connected layer, the local feature extraction layer is connected between the downsampling layer and the fully connected layer, the global feature extraction layer is connected between the downsampling layer and the fully connected layer, and the global feature extraction layer is connected in parallel with the local feature extraction layer. It can be seen that the first image is input into the downsampling layer as an input item, and the feature image is output through the downsampling layer; the feature image of the downsampling layer is respectively input into the local feature extraction layer and the global feature extraction layer, the local feature extraction layer extracts the local features of the feature image, and the global feature extraction layer extracts the global features of the feature image; the local features output by the local feature extraction layer and the global features output by the global feature extraction layer are respectively input into the fully connected layer, so as to output the bilateral grid corresponding to the first image through the fully connected layer. In addition, in a possible implementation of the present embodiment, the downsampling layer includes a downsampling convolution layer and four first convolution layers, the convolution kernel of the first convolution layer is 1*1, and the step size is 1; the local feature extraction layer may include two second convolution layers, the convolution kernels of the two second convolution layers are both 3*3, and the step size is 1; the global feature extraction layer may include two third convolution layers and three fully connected layers, the convolution kernels of the two third convolution layers are both 3*3, and the step size is 2.

[0146] Further, the convolution unit 30 includes a fourth convolution layer, the first image is input into the fourth convolution layer, and the guidance image is input into the fourth convolution layer, wherein the guidance image has the same resolution as the first image. For example, the first image is a color image, and the fourth convolution layer performs a pixel-level operation on the first image so that the guidance image is a grayscale image of the first image.

[0147] For example: the first image I is input into the downsampling convolution layer, and the downsampling convolution layer outputs a 256x256 three-channel low-resolution image. The 256x256 three-channel low-resolution image passes through four first convolution layers in sequence to obtain a 16x16 64-channel feature image; the 16x16 64-channel feature image is input into the local feature extraction layer to obtain the local feature L, and the 16x16 64-channel feature image is input into the global feature extraction layer to obtain the global feature; the local features and global features are input into the fully connected layer, and the bilateral grid is output through the fully connected layer. In addition, the first image is input into the convolution unit, and the guidance image corresponding to the first image is input into the convolution unit.

[0148] Further, in an implementation of this embodiment, the transformation module 200 includes a segmentation unit 40 and a transformation unit 50. Accordingly, Fig.11 , 12 As shown in 15, the step of inputting the guiding image, the bilateral grid and the first image into the transformation module, and generating a generated image corresponding to the first image through the transformation module specifically includes:

[0149] M121, inputting the guidance image into the segmentation unit, and segmenting the bilateral grid by the segmentation unit to obtain a color transformation matrix of each pixel in the first image;

[0150] M122. Input the first image and the color transformation matrix of each pixel in the first image into the transformation unit, and generate a generated image corresponding to the first image through the transformation unit.

[0151] Specifically, the segmentation unit 40 includes an upsampling layer, and the input items of the upsampling layer are the guidance image and the bilateral grid. The bilateral grid is upsampled by the guidance image to obtain the color transformation matrix of each pixel in the first image. The upsampling process of the upsampling layer can be upsampling the bilateral grid with reference to the guidance image to obtain the color transformation matrix of each pixel in the first image. In addition, the input items of the transformation unit 60 are the color transformation matrix of each pixel and the first image. The color of the corresponding pixel in the first image is transformed by the color transformation matrix of each pixel to obtain the generated image corresponding to the first image.

[0152] Further, in the step M20, the preset condition includes that the loss function value meets the preset requirement or the number of trainings reaches the preset number. The preset requirement can be determined according to the accuracy of the first image processing model, which is not described in detail here. The preset number can be the maximum number of trainings of the first preset network model, for example, 5000 times. Thus, the first preset network model outputs a generated image, and the loss function value of the first preset network model is calculated according to the generated image and the second image. After the loss function value is calculated, it is determined whether the loss function value meets the preset requirement; if the loss function value meets the preset requirement, the training is terminated; if the loss function value does not meet the preset requirement, it is determined whether the number of trainings of the first preset network model reaches the prediction number. If the preset number is not reached, the network parameters of the first preset network model are corrected according to the loss function value; if the preset number is reached, the training is terminated. In this way, judging whether the training of the first preset network model is terminated by the loss function value and the number of trainings can avoid the training of the first preset network model entering an infinite loop due to the failure of the loss function value to meet the preset requirement.

[0153] Furthermore, since the modification of the network parameters of the first preset network model is performed when the training situation of the first preset network model does not meet the preset conditions (i.e., the loss function value does not meet the preset requirements and the number of training times has not reached the preset number of times), after the network parameters of the first preset network model are corrected according to the loss function value, it is necessary to continue to train the network model, that is, continue to execute the step of inputting the first image in the training image set into the first preset network model. Among them, the first image that continues to be input into the first preset network model of the first image in the training image set is the first image that has not been input into the first preset network model as an input item. For example, all first images in the training image set have a unique image identifier (e.g., an image number), and the image identifier of the first image input for the first training is different from the image identifier of the first image input for the second training, such as the image number of the first image input for the first training is 1, the image number of the first image input for the second training is 2, and the image number of the first image input for the Nth training is N. Of course, in practical applications, since the number of first images in the training image set is limited, in order to improve the training effect of the first image processing model, the first images in the training image set can be input into the first preset network model in sequence to train the first preset network model. After all the first images in the training image set have been input into the first preset network model, the operation of inputting the first images in the training image set into the first preset network model in sequence can continue to be performed, so that the training image group in the training image set is input into the first preset network model in a cycle.

[0154] In addition, the degree of diffusion of the highlight part of the image captured at different exposures is different, so the degree of diffusion of the highlight part of the image captured by the under-screen imaging system at different light intensities is different, which makes the image quality captured by the under-screen imaging system different. Therefore, when training the first image processing model, multiple training image sets can be obtained, each training image set corresponds to a different exposure, and each training image set is used to train the first preset network model to obtain the model parameters corresponding to each training image set. In this way, using the first image with the same exposure as the training sample image can increase the training speed of the network model, and at the same time make different exposures correspond to different model parameters. When the first image processing model is used to process the image to be processed with color cast, the corresponding model parameters can be selected according to the exposure corresponding to the denoised image, and the diffusion of the highlight part of the image at each exposure is suppressed to improve the image quality of the processed image corresponding to the denoised image.

[0155] Furthermore, in an implementation of the present embodiment, the training image set includes several training sub-image sets, each training sub-image set includes several groups of training sample image groups, the exposure of the first image in any two groups of training sample image groups in the several training image groups is the same (that is, for each group of training image groups, the exposure of the first image in each group of training sample image groups in the group is the same), the exposure of the second image in each group of training sample image groups in the several training image groups is within a preset range, and the exposure of the first image in any two training sub-image sets is different. Wherein, the preset range of the exposure of the second image can be determined according to the exposure time and ISO (the aperture of the existing mobile phone is a fixed value), the preset range of the exposure represents the exposure of the image taken without exposure compensation, and the second image captured by the on-screen camera at the first exposure within the preset range of the exposure is a normal exposure image. By adopting the normal exposure image as the second image, the image output by the first image processing model trained according to the training image set can have a normal exposure, so that the first image processing model has a brightening function. For example, when the image A input to the image processing model is a low-exposure image, after image A is processed by the first image processing model, the exposure of the output image A can be made normal, thereby improving the image brightness of image A.

[0156] For example: assume that the exposure of an image includes 5 levels, which are recorded as 0, -1, -2, -3 and -4, wherein the exposure increases as the exposure level decreases. For example, the exposure corresponding to exposure level 0 is lower than the exposure corresponding to exposure level -4. The training image set may include five training sub-image sets, which are respectively recorded as a first training sub-image set, a second training sub-image set, a third training sub-image set, a fourth training sub-image set and a fifth training sub-image set. In each group of training image groups included in the first training sub-image set, the exposure of the first image corresponds to level 0, and the second image is an image with an exposure within a preset range; in each group of training image groups included in the second training sub-image set, the exposure of the first image corresponds to level -1, and the second image is an image with an exposure within a preset range; in each group of training image groups included in the third training sub-image set, the exposure of the first image corresponds to level -2, and the second image is an image with an exposure within a preset range; in each group of training image groups included in the fourth training sub-image set, the exposure of the first image corresponds to level -3, and the second image is an image with an exposure within a preset range; in each group of training image groups included in the fifth training sub-image set, the exposure of the first image corresponds to level -4, and the second image is an image with an exposure within a preset range. Of course, it is worth noting that the number of training image groups included in the first training sub-image set, the second training sub-image set, the third training sub-image set, the fourth training sub-image set, and the fifth training sub-image set may be the same or different. For example, the first training sub-image set, the second training sub-image set, the third training sub-image set, the fourth training sub-image set, and the fifth training sub-image set all include 5,000 training image groups.

[0157] In addition, for each training sub-image set, the training sub-image set is a training image set of the first preset network model, and the first preset network model is trained through the training sub-image set to obtain the model parameters corresponding to the training sub-image set. Wherein, the process of training the first preset network model with the training sub-image set as a training image set includes: the first preset network model generates a generated image corresponding to the first image according to the first image in the training sub-image set; the first preset network model modifies the model parameters according to the second image corresponding to the first image and the generated image corresponding to the first image, and the first preset network model continues to execute the step of generating the generated image corresponding to the first image according to the first image in the training sub-image set, until the training situation of the first preset network model meets the preset conditions to obtain the model parameters corresponding to the training sub-image, which can be specifically parameterized in step M10 and step M20, which will not be repeated here.

[0158] Furthermore, the training process of each training sub-image set for the first preset network model is independent of each other, that is, each training sub-image set is used to train the first preset network model. At the same time, a number of model parameters can be obtained by training the first preset network model with a training sub-image set, each model parameter is obtained by training with a training sub-image set, and the training sub-image sets corresponding to any two model parameters are different. It can be seen that the first image processing model corresponds to a number of model parameters, and the model parameters correspond to a number of training sub-image sets one by one.

[0159] For example: taking the above training sample images including the first training sub-image set, the second training sub-image set, the third training sub-image set, the fourth training sub-image set and the fifth training sub-image set as an example, the first image processing model includes 5 model parameters, which are respectively recorded as the first model parameter, the second model parameter, the third model parameter, the fourth model parameter and the fifth model parameter, wherein the first model parameter corresponds to the first training sub-image set, the second model parameter corresponds to the second training sub-image set, the third model parameter corresponds to the third training sub-image set, the fourth model parameter corresponds to the fourth training sub-image set, and the fifth model parameter corresponds to the fifth training sub-image set.

[0160] Further, when the training image set includes several training sub-image sets, the first preset network model is trained according to each training sub-image set. Here, the training image set includes 5 training sub-image sets as an example for explanation. The process of using the first training sub-image set, the second training sub-image set, the third training sub-image set, the fourth training sub-image set and the fifth training sub-image set to train the first preset network model respectively can be: firstly, the first training sub-image set is used to train the first preset network model to obtain the first model parameters corresponding to the first training sub-image set, and then the second training sub-image set is used to train the first preset network model to obtain the second model parameters corresponding to the second training sub-image set, and so on to obtain the fifth model parameters corresponding to the fifth training sub-image set.

[0161] In addition, when the same first preset network model is used to train multiple training sub-image sets separately, there will be a problem that each training sub-image set affects the model parameters of the first preset network model. For example, assuming that training sub-image set A includes 1,000 training image groups and training sub-image set B includes 200 training image groups, then the model parameters corresponding to the training sub-image set B obtained by first training the first preset network model with training sub-image set A and then training the first preset network model with training sub-image set B are different from the model parameters corresponding to the training sub-image set B obtained by training the first preset network model only with training sub-image set B.

[0162] Therefore, in one implementation of the present embodiment, after training a training sub-image set, the first preset network model can be initialized first, and then the initialized first preset network model can be used to train the next training sub-image set. For example, after the first preset network model is trained according to the first training sub-image set and the first model parameters corresponding to the first training sub-image set are obtained, the first preset network model can be initialized so that the initial model parameters and model structure of the first preset network model used to train the second model parameters are the same as the first preset network model used to train the first model parameters. Of course, before training the third model parameters, the fourth model parameters and the fifth model parameters, the first preset network model can be initialized so that the initial model parameters and model structure of the first preset network model corresponding to each training sub-image set are the same. Of course, in actual applications, after the first preset network model is trained according to the first training sub-image set to obtain the first model parameters corresponding to the first training sub-image set, the second training sub-image set can be directly trained using the first preset network model (configured with the first model parameters) trained based on the first training sub-image set to obtain the second model parameters corresponding to the second training sub-image set, and the step of training the first preset network model (configured with the second model parameters) according to the third training sub-image set is continued until the fifth training sub-image set is trained to complete the fifth model parameters corresponding to the fifth training sub-image set.

[0163] In addition, the first training sub-image set, the second training sub-image set, the third training sub-image set, the fourth training sub-image set, and the fifth training sub-image set all include a certain number of training image groups, so that each group of training sub-images can meet the training requirements of the first preset network model. Of course, in actual applications, when the first preset network model is trained based on each training sub-image set, the training image groups in the training sub-image set can be cyclically input into the first preset network model to train the first preset network model so that the first preset network model meets the preset requirements.

[0164] Further, in an implementation of the present embodiment, the acquisition process of the training samples including each training sub-image set may be: first, the under-screen imaging system is set to the first exposure, the first image in the first training sub-image set is acquired through the under-screen imaging system, and the second image corresponding to the first image in the first training sub-image set is acquired through the on-screen imaging system; after the first training sub-image set is acquired, the under-screen imaging system is set to the second exposure, and the first image and the second image corresponding to the first image in the second training sub-image set are acquired through the under-screen imaging system and the on-screen imaging system; after the second training sub-image set is acquired; continue to perform the steps of setting the exposure of the under-screen imaging system and acquiring the training sub-image set until all the training sub-image sets included in the training image set are acquired. Among them, the number of training image groups included in each training sub-image set included in the training image set may be the same or different. In an implementation of the present embodiment, the number of training image groups included in each training sub-image set included in the training image set may be the same, for example, the number of training image groups included in each training sub-image set is 5000.

[0165] Furthermore, since each training sub-image set corresponds to a different exposure, after obtaining the model parameters corresponding to each training sub-image set, for each training sub-image set, the model parameters corresponding to the training sub-image set can be associated with the exposure corresponding to the training sub-image set to establish a corresponding relationship between the exposure and the model parameters. In this way, when the denoised image is processed by the first image processing model, the exposure of the denoised image can be obtained first, and then the model parameters corresponding to the denoised image can be determined according to the exposure, and then the model parameters corresponding to the denoised image can be configured in the first preset network model to obtain the first image processing model corresponding to the denoised image, so as to use the first image processing model to process the denoised image. In this way, for denoised images with different exposures, the first image processing model configured with different network parameters can be determined, and the denoised image can be processed by the first image processing model corresponding to the denoised image, avoiding the influence of the exposure on the color cast, thereby improving the effect of removing the color cast of the denoised image. In addition, the second image can be a normal exposure, so that the output image output by the first image processing model is a normal exposure, which has a brightening effect on the denoised image.

[0166] Furthermore, it can be known from the generation process of the first image processing model that in a possible implementation of this embodiment, the first image processing model may include a number of model parameters, and each model parameter corresponds to an exposure level. Therefore, in this implementation, after obtaining the denoised image, the number of model parameters included in the first image processing model may be detected first. When the number of model parameters is one, the denoised image may be directly input into the first image processing model to process the denoised image through the image processing; when there are multiple model parameters, the exposure level of the denoised image may be obtained first, and then the model parameters corresponding to the denoised image may be determined according to the exposure level, and the model parameters corresponding to the denoised image may be configured in the first image processing model to update the model parameters configured by the image processing parameters, and the denoised image may be input into the updated first image processing model.

[0167] Furthermore, in an implementation of this embodiment, the first image processing model corresponds to a plurality of model parameters, each of which is obtained by training based on a training sub-image set, and the training sub-image sets corresponding to any two model parameters are different from each other (for example, the training sub-image set corresponding to model parameter A is different from the training sub-image set corresponding to model parameter B). Accordingly, the step of inputting the denoised image to the trained first image processing model specifically includes:

[0168] A101. Extract the exposure of the denoised image.

[0169] Specifically, the exposure is the degree to which the photosensitive element of the image acquisition device is irradiated by light, which is used to reflect the exposure degree during imaging. The denoised image can be an RGB three-channel image, and the exposure of the denoised image is determined according to the highlight area of ​​the denoised image, and at least one of the R (i.e., red channel) value, G (i.e., green channel) value, and B (i.e., blue channel) value of each pixel point contained in the highlight area is greater than a preset threshold. Of course, in practical applications, the denoised image can also be a Y channel image or a Bayer format image, and when the denoised image is a Y channel image or a Bayer format image (Raw format), before extracting the denoised image, the Y channel image or the Bayer format image needs to be converted into an RGB three-channel image, so as to determine the highlight area of ​​the denoised image according to the red channel R value, green channel G value, and blue channel B value of the denoised image.

[0170] Furthermore, in an implementation of this embodiment, extracting the exposure of the denoised image specifically includes:

[0171] H10, determining a first pixel point that meets a preset condition according to the red channel R value, the green channel G value, and the blue channel B value of each pixel point in the denoised image, wherein the preset condition is that at least one of the R value, the G value, and the B value is greater than a preset threshold;

[0172] H20. Determine a highlight area of ​​the denoised image according to all first pixel points that meet a preset condition, and determine an exposure of the denoised image according to the highlight area.

[0173] Specifically, the denoised image is an RGB three-channel image, so that for each pixel in the denoised image, the pixel includes a red channel R value, a green channel G value and a blue channel B value, that is, for each pixel in the denoised image, the red channel R value, the green channel G value and the blue channel B value of the pixel can be obtained. Therefore, in the process of extracting the exposure of the denoised image, firstly, for each pixel of each denoised image, the red channel R value, the green channel G value and the blue channel B value of the pixel are obtained, and then the R value, the G value and the B value of each pixel are respectively compared with the preset threshold value to obtain the first pixel in the denoised image that meets the preset condition. The preset condition is that at least one of the R value, G value and B value is greater than a preset threshold value, and the first pixel point satisfies the preset condition means that the R value of the first pixel point is greater than the preset threshold value, the G value of the first pixel point is greater than the preset threshold value, the B value of the first pixel point is greater than the preset threshold value, the R value and G value of the first pixel point are both greater than the preset threshold value, the R value and B value of the first pixel point are both greater than the preset threshold value, the G value and B value of the first pixel point are both greater than the preset threshold value, or the R value, B value and G value of the first pixel point are all greater than the preset threshold value.

[0174] Further, after obtaining all the first pixel points that meet the preset conditions, all the first pixel points obtained are recorded as a first pixel point set. There are adjacent pixel points and non-adjacent pixel points in the first pixel point set, wherein the pixel adjacency refers to the adjacent positions of the pixel points in the denoised image, and the non-adjacent positions refers to the non-adjacent positions of the pixel points in the denoised image, and the adjacent positions refer to that in the pixel coordinates to be processed, one of the horizontal coordinates and the vertical coordinates of two adjacent pixel points is the same. For example, the first pixel point set includes pixel point (100, 101), pixel point (100, 100), pixel point (101, 101) and pixel point (200, 200). Then pixel point (100, 101) and pixel point (100, 100) are adjacent pixel points, and pixel point (100, 101) and pixel point (101, 101) are adjacent pixel points, while pixel point (100, 101), pixel point (100, 100), pixel point (101, 101) and pixel point (200, 200) are all non-adjacent pixel points.

[0175] Furthermore, the highlight region is a connected region formed by adjacent pixels in the first pixel set, that is, the pixel value of each first pixel included in the highlight region satisfies a preset condition. Therefore, in an implementation of this embodiment, determining the highlight region of the denoised image based on all first pixels that meet the preset condition specifically includes:

[0176] L10, obtaining the connected areas formed by all the first pixel points that meet the preset conditions, and selecting the target area that meets the preset rules from all the obtained connected areas, wherein the preset rule is that the R value, G value and / or B value of the first pixel points in the target area that are greater than the preset threshold are of the same type;

[0177] L20. Calculate the area corresponding to each target area obtained by screening, and select the target area with the largest area as the highlight area.

[0178] Specifically, the connected region is a closed region formed by all adjacent first pixel points in the first pixel point set, each pixel point included in the connected region is a first pixel point, and for each first pixel point A in the connected region, at least one first pixel point B in the connected region is adjacent to the first pixel point A. At the same time, for each first pixel point C in the first pixel point set except the first pixel point included in the connected region, the first pixel point C is not adjacent to any first pixel point A in the connected region. For example, the first pixel point set includes pixel point (100, 101), pixel point (100, 100), pixel point (101, 100), pixel point (101, 101), pixel point (100, 102) and pixel point (200, 200), then pixel point (100, 101), pixel point (100, 100), pixel point (101, 100), pixel point (101, 101), pixel point (100, 102) form a connected region.

[0179] In addition, since the connected areas of the denoised image are formed by a light source, and the light source generates light of the same color. Therefore, after obtaining all the connected areas contained in the denoised image, the connected areas can be selected according to the area colors corresponding to each connected area. Therefore, after obtaining the connected areas of the denoised image, it is determined whether the types of the R values, G values ​​and / or B values ​​of the first pixel points in the connected area that are greater than the preset threshold are the same, so as to determine whether the connected area meets the preset rules. The same type means that for two first pixel points, respectively recorded as pixel point A and pixel point B, if the R value of pixel point A is greater than the preset threshold, then the pixel point B also has only the R value greater than the preset threshold; if the R value and G value of pixel point A are both greater than the preset threshold, then the pixel point B also has only the R value and G value greater than the preset threshold; if the R value, G value and B value of pixel point A are all greater than the preset threshold, then the R value, G value and B value of pixel point B are all greater than the preset threshold. The different types refer to that for two first pixels, respectively recorded as pixel C and pixel D, if the V value of pixel C (the V value can be one of the R value, G value, and B value) is greater than the preset threshold, then the V value of pixel D is less than or equal to the preset threshold, or the V value of pixel D is greater than the preset threshold and at least one M value (the M value is one of the two values ​​of R value, G value, and B value except the V value) is greater than the preset threshold. For example, if the R value of pixel C is greater than the preset threshold, and the R value of pixel D is less than or equal to the preset threshold, then the types of pixel C and pixel D are different; for another example, if the R value of pixel C is greater than the preset threshold, the R value of pixel D is greater than the preset threshold, and the G value of pixel D is greater than the preset threshold, then the types of pixel C and pixel D are different. In this embodiment, the preset rule is that the types of the R value, G value, and / or B value of the first pixel in each connected area that are greater than the preset threshold are the same.

[0180] Furthermore, since the denoised image may include multiple target areas, after the target areas are obtained, the target areas can be screened according to their areas to obtain highlight areas. The area of ​​the target area refers to the area of ​​the target area in the denoised image, and the area is calculated in the pixel coordinate system of the denoised image. After the areas of the target areas are obtained, the areas of the target areas can be compared, and the target area with the largest area can be selected as the highlight area. By using the target area with the largest area as the highlight area, the area with the largest brightness area in the denoised image can be obtained, and the exposure can be determined according to the area with the largest brightness area, which can improve the accuracy of the exposure.

[0181] Further, in an implementation of this embodiment, determining the exposure of the denoised image according to the highlight area specifically includes:

[0182] P10, calculating a first area of ​​the highlight region and a second area of ​​the denoised image;

[0183] P20. Determine an exposure level corresponding to the denoised image according to a ratio of the first area to the second area.

[0184] Specifically, the second area of ​​the denoised image is calculated based on the image size of the denoised image. For example, if the image size of the denoised image is 400*400, then the image area of ​​the denoised image is 400*400=160000. The first area of ​​the highlight region is the area of ​​the highlight region in the pixel coordinate system of the denoised image. For example, if the highlight region is a square region with a side length of 20, then the first area of ​​the highlight region is 20*20=400.

[0185] Furthermore, in order to determine the exposure according to the ratio of the first area to the second area, the corresponding relationship between the ratio interval and the exposure is preset. After the ratio is obtained, the ratio area where the ratio is located is first obtained, and the exposure corresponding to the ratio interval is determined according to the corresponding relationship to obtain the exposure of the denoised image. For example, the corresponding relationship between the ratio interval and the exposure is: when the interval is [0, 1 / 100), the exposure corresponds to level 0; when the interval is [1 / 100, 1 / 50), the exposure corresponds to level -1; when the interval is [1 / 50, 1 / 20), the exposure corresponds to level -2; when the interval is [1 / 20, 1 / 50), the exposure corresponds to level -3; when the interval is [1 / 20, 1], the exposure corresponds to level -4. Then, when the ratio of the first area to the second area is 1 / 10, the ratio is in the interval [1 / 20, 1], so that the exposure corresponding to the denoised image is level -4.

[0186] A102. Determine model parameters corresponding to the denoised image according to the exposure, and use the model parameters to update model parameters of the first image processing model.

[0187] Specifically, a correspondence between exposure and model parameters is established during the training of the first image processing model, so that after the exposure of the denoised image is obtained, the model parameters corresponding to the exposure can be determined according to the correspondence between the exposure and the model parameters, wherein the exposure refers to the exposure level, that is, the correspondence between the exposure and the model parameters is the correspondence between the exposure level and the model parameters. In addition, it can be known from the above that each exposure level corresponds to a ratio interval, so after the denoised image is obtained, the ratio of the area of ​​the highlight area in the denoised image to the image area can be obtained, and the ratio interval in which the ratio is located can be determined, and then the exposure level corresponding to the denoised image can be determined according to the ratio area, and finally the model parameters corresponding to the denoised image can be determined according to the exposure level, so as to obtain the model parameters corresponding to the denoised image. In addition, after the model parameters corresponding to the exposure are obtained, the model parameters configured by the first image processing model are updated using the obtained model parameters to update the first image processing model, that is, the first image processing model corresponding to the obtained model parameters.

[0188] A103. Input the denoised image into the updated first image processing model.

[0189] Specifically, the denoised image is used as an input item of the updated first image processing model, and the denoised image is output to the updated first image processing model to process the denoised image. It can be understood that the model parameters of the image processing model corresponding to the image to be processed are model parameters determined according to the exposure of the image to be processed, and the model parameters are model parameters obtained by training the preset network model, so that the accuracy of the updated image processing model in processing the image to be processed can be guaranteed.

[0190] Further, in an implementation of this embodiment, generating an output image corresponding to the denoised image through the first image processing model refers to inputting the denoised image into the first image processing model as an input item of the first image processing model, adjusting the image color of the denoised image through the first image processing model to obtain an output image, wherein the output image is an image corresponding to the image to be denoised after decolorization processing. For example, Fig.16 The denoised image shown in FIG. 1 is obtained by processing the image as described above. Fig.17 The output image is shown.

[0191] Furthermore, it can be known from the training process of the first image processing model that the first image processing model includes a downsampling module and a transformation module, so when the first image processing model is used to process the image to be processed, it is necessary to process it in sequence through the downsampling module and the transformation module. Accordingly, the first image processing model includes; the output image corresponding to the denoised image generated by the first image processing model specifically includes:

[0192] A201, inputting the denoised image into the downsampling module, obtaining a bilateral grid corresponding to the image to be processed and a guidance image corresponding to the image to be processed through the downsampling module, wherein the resolution of the guidance image is the same as the resolution of the image to be processed;

[0193] A202. Input the guiding image, the bilateral grid and the denoised image into the transformation module, and generate an output image corresponding to the denoised image through the transformation module.

[0194] Specifically, the input item of the downsampling module is the denoised image, and the output item is the bilateral grid and the guidance image corresponding to the image to be denoised. The input item of the transformation module is the guidance image, the bilateral grid and the image to be processed, and the output item is the output image. Among them, the structure of the downsampling module is the same as the structure of the downsampling module in the first preset network model, and the specific description of the structure of the downsampling module in the first preset network model can be referred to. The processing of the image to be processed by the downsampling module of the first image processing model is the same as the processing process of the first image by the downsampling module in the first preset network model, so the specific execution process of the step A201 can refer to step M11. Similarly, the structure of the transformation module is the same as the structure of the transformation module in the first preset network model, and the specific description of the structure of the transformation module in the first preset network model can be referred to. The processing of the image to be processed by the transformation module of the first image processing model is the same as the processing process of the first image by the transformation module in the first preset network model, so the specific execution process of the step A202 can refer to step M12.

[0195] Further, in an implementation of this embodiment, the downsampling module includes a downsampling unit and a convolution unit. Accordingly, the step of inputting the denoised image into the downsampling module and obtaining the bilateral grid corresponding to the denoised image and the guidance image corresponding to the image to be processed through the downsampling module specifically includes:

[0196] A2011, inputting the denoised image into the downsampling unit and the convolution unit respectively;

[0197] A2012. Obtain a bilateral grid corresponding to the denoised image through the downsampling unit, and obtain a guidance image corresponding to the image to be processed through the convolution unit.

[0198] Specifically, the input item of the downsampling unit is the denoised image, and the output item is the bilateral grid, and the input item of the convolution unit is the denoised image, and the output item is the guidance image. Among them, the structure of the downsampling unit is the same as the structure of the downsampling unit in the first preset network model, and the specific description of the structure of the downsampling unit in the first preset network model can be referred to. The processing of the image to be processed by the downsampling unit of the first image processing model is the same as the processing process of the first image by the downsampling unit in the first preset network model, so the specific execution process of the step A2011 can refer to step M111. Similarly, the structure of the convolution unit is the same as the structure of the convolution unit in the first preset network model, and the specific description of the structure of the convolution unit in the first preset network model can be referred to. The processing of the denoised image by the convolution unit of the first image processing model is the same as the processing process of the first image by the convolution unit in the first preset network model, so the specific execution process of the step A2012 can refer to step M112.

[0199] Furthermore, in an implementation of this embodiment, the transformation module includes a segmentation unit and a transformation unit. Accordingly, the step of inputting the guide image, the bilateral grid, and the image to be processed into the transformation module, and generating an output image corresponding to the denoised image through the transformation module specifically includes:

[0200] A2021, inputting the guiding image into the segmentation unit, and segmenting the bilateral grid by the segmentation unit to obtain a color transformation matrix of each pixel in the image to be processed;

[0201] A2022. Input the color transformation matrix of each pixel in the denoised image and the image to be processed into the transformation unit, and generate an output image corresponding to the denoised image through the transformation unit.

[0202] Specifically, the input items of the segmentation unit are the guidance image and the bilateral grid, the output item is the color transformation matrix of each pixel in the image to be processed, the input items of the transformation unit are the denoised image and the color transformation matrix of each pixel in the denoised image, and the output item is the output image. Among them, the structure of the segmentation unit is the same as the structure of the segmentation unit in the first preset network model, and the specific description of the structure of the segmentation unit in the first preset network model can be referred to. The processing of the bilateral grid and the guidance image corresponding to the image to be processed by the segmentation unit of the first image processing model is the same as the processing process of the bilateral grid and the guidance image corresponding to the first image by the downsampling unit in the first preset network model, so the specific execution process of the step A2021 can refer to step M121. Similarly, the structure of the transformation unit is the same as the structure of the transformation unit in the first preset network model, and the specific description of the structure of the transformation unit in the first preset network model can be referred to. The transformation unit of the first image processing model processes the image to be processed based on the color transformation matrix of each pixel in the image to be processed, which is the same as the processing process of the first image by the transformation unit in the first preset network model based on the color transformation matrix of each pixel in the first image. Therefore, the specific execution process of step A2022 can refer to step M122.

[0203] It is understandable that the network structure corresponding to the first image processing model during the training process is the same as the network structure corresponding to the application process (removing the color cast carried by the image to be processed). For example, during the training process, the first image processing model includes a downsampling module and a transformation module, then correspondingly, when the denoised image is de-colorized by the first image processing model, the first image processing model also includes a downsampling module and a transformation module.

[0204] For example, during the training process, the downsampling module of the first image processing model includes a downsampling unit and a convolution unit, and the transformation module includes a segmentation unit and a transformation unit; accordingly, when the denoised image is decolorized through the first image processing model, the downsampling module may also include a downsampling unit and a convolution unit, and the transformation module includes a segmentation unit and a transformation unit; and during the application process, the working principle of each layer is the same as the working principle of each layer during the training process. Therefore, the input and output conditions of each layer of the neural network during the application process of the first image processing model can be referred to the relevant introduction during the training process of the first image processing model, and will not be repeated here.

[0205] Compared with the prior art, the present invention provides an image processing method, a storage medium and a terminal device, wherein the image processing method comprises obtaining a set of images to be processed, generating a denoised image corresponding to the set of images to be processed according to the set of images to be processed, and inputting the denoised image into a first image processing model that has been trained, and generating an output image corresponding to the denoised image through the first image processing model. The present invention first obtains a plurality of images, and generates a denoised image according to the plurality of images, and then uses a first image processing model that has been trained to adjust the image color of the denoised image by deep learning based on the training image set, so that the color quality and noise quality of the output image can be improved, thereby improving the image quality.

[0206] S40, inputting the processed image into a trained second image processing model, and performing deghosting processing on the processed image through the second image processing model to obtain an output image.

[0207] Specifically, the second image processing model may be pre-trained by an image device (e.g., a mobile phone equipped with an under-screen camera) that processes the image set to be processed, or the file corresponding to the second image processing model may be transplanted to the image device after training. In addition, the image device may use the second image processing model as a de-ghosting function module. When the image device obtains the processed image, the de-ghosting function module is activated to output the image to be processed to the second image processing model.

[0208] Further, in one implementation of this embodiment, Fig.18 As shown, the training process of the second image processing model may include:

[0209] L100. The second preset network model generates a generated image corresponding to the third image according to the third image in the training image set.

[0210] Specifically, the second preset network model can adopt a deep learning network model, the training image set includes multiple groups of training image groups with different image contents, each group of training image groups includes a third image and a fourth image, the third image corresponds to the fourth image, they present the same image scene, the fourth image is a normally displayed image (i.e., the original image), the image content of the third image corresponds to the fourth image, but the object in the image content has a ghost or a blur effect similar to a ghost. Wherein, the ghost refers to the formation of a virtual image around the object in the image, for example, it can include the situation where one or more contours or virtual images appear at the edge of the object in the image. For example, when a double image appears on the object in the image (i.e., a contour or virtual image appears at the edge of the object), the image column with smaller pixel values ​​can be understood as the real image of the object, and the image column with larger pixel values ​​can be understood as the contour or virtual image of the object.

[0211] Further, the third image and the fourth image correspond to the same image scene. The third image and the fourth image correspond to the same image scene means that the similarity between the image content carried by the third image and the image content carried by the fourth image reaches a preset threshold, and the image size of the third image is the same as the image size of the fourth image, so that when the third image and the fourth image overlap, the coverage rate of the object carried by the third image to the corresponding object in the fourth image reaches a preset condition. Among them, the preset threshold can be 99%, the preset condition can be 99.5%, etc.

[0212] In addition, in an implementation of the present embodiment, in order to reduce the impact of the image difference between the third image and the fourth image on the training of the second preset network model, the image content of the third image and the image content of the fourth image can be exactly the same. For example, the third image is an image with a ghosting image of an image size of 600*800, the image content of the third image is a square, and the positions of the four vertices of the square in the third image are (200, 300), (200, 400), (300, 400) and (300, 300) respectively. Then, the image size of the fourth image is an image without ghosting of 600*800, the image content of the fourth image is a square, and the positions of the four vertices of the square in the fourth image are (200, 300), (200, 400), (300, 400) and (300, 300) respectively. When the third image is placed on the fourth image and overlaps with the fourth image, the third image covers the fourth image, and the square in the third image overlaps with the square in the fourth image up and down.

[0213] Furthermore, the fourth image may be an image obtained by normal shooting, such as an image captured by an under-screen camera after removing the display panel in the under-screen imaging system, or an image captured by an under-screen camera by making an experimental display panel without light-shielding structures such as data lines and scan lines to replace the actual display panel, and then using it as the display panel of the under-screen imaging system. It may also be an image obtained through the Internet (such as Baidu), or it may be an image sent by other external devices (such as smart phones). The third image may be captured by an under-screen imaging system (for example, an under-screen camera), or it may be obtained by processing the fourth image. The processing of the fourth image refers to forming a ghost image on the fourth image. In a possible implementation, the image size and image content of the fourth image may be kept unchanged during the processing.

[0214] In an implementation of this embodiment, the third image is captured by an under-screen imaging system, the third image and the fourth image have the same shooting parameters, and the shooting scene corresponding to the third image is the same as the shooting scene of the fourth image. Fig.21 The image shown in FIG. 1 is obtained by shooting a scene with the under-screen camera. The image content is blurred due to the influence of the shading structure in the display panel. The fourth image is as shown in FIG. Fig. 22 The normally displayed image shown. Meanwhile, in a possible implementation of the embodiment, the shooting parameters may include exposure parameters of the imaging system, wherein the exposure parameters may include aperture, shutter speed, sensitivity, focus, and white balance, etc. Of course, in practical applications, the shooting parameters may also include ambient light, shooting angle, and shooting range, etc.

[0215] Furthermore, when the third image is an image captured by an under-screen imaging system, since the third image and the fourth image may be captured by two different imaging systems, when the imaging system is replaced, the shooting position or shooting angle may change, so that the third image and the fourth image are not aligned in space. Therefore, before the second preset network model generates a generated image corresponding to the third image according to the third image in the training image set, it also includes:

[0216] M10. For each training image group in the training image set, align the third image in the training image group with the fourth image corresponding to the third image to obtain an aligned image aligned with the fourth image, and use the aligned image as the third image.

[0217] Specifically, the alignment process for each group of training image groups in the training image set refers to performing alignment processing on each group of training image groups in the training image set. The alignment processing can be that after acquiring the training image set, each group of training image groups is aligned to obtain an aligned training image group, and after all groups of training image groups are aligned, the step of inputting the third image in each group of training image groups into the second preset network model is performed; of course, it can also be that before the third image in each group of training image groups is input into the second preset network model, the group of training image groups is aligned to obtain the aligned training image group corresponding to the group of training images, and then the third image in the aligned training image group is input into the second preset network model. In this embodiment, the alignment processing is performed on each group of training image groups after acquiring the training image set, and after all training image groups have completed the alignment processing, the operation of inputting the third image in the training image set into the second preset network model is performed. In addition, it is worth mentioning that the process of aligning the third image with the fourth image corresponding to the third image is the same as the process of aligning the reference image with the benchmark image. Here, the third image is used as the reference image and the fourth image is used as the benchmark image. Therefore, the process of aligning the third image with the fourth image will not be described in detail here. For details, please refer to the description of the process of aligning the reference image with the benchmark image mentioned above.

[0218] Further, in one implementation of this embodiment, Fig.19 As shown, the second preset network model includes an encoder and a decoder; the second preset network model generates a generated image corresponding to the third image according to the third image in the training image set, specifically including:

[0219] L101, inputting the third image in the training image set into the encoder, and obtaining a feature image of the third image through the encoder, wherein the image size of the feature image is smaller than the image size of the third image;

[0220] L102. Input the feature image into the decoder, and output the generated image through the decoder, wherein the image size of the generated image is equal to the image size of the third image.

[0221] Specifically, the second preset network model adopts a decoding-encoding structure, and the decoding-encoding structure is a convolutional neural network CNN structure, wherein the encoder 1000 is used to convert the input image into a feature image whose image space size is smaller than the input image and the number of channels is greater than the input image, and the decoder 2000 is used to convert the feature image into a generated image with the same image size as the input image. In this embodiment, the encoder includes a first redundant learning layer 101 and a downsampling layer 102 arranged in sequence, the third image in the training image group is input to the first redundant learning layer 101, and the first redundant learning layer 101 outputs a first feature map with the same image size as the third image; the first feature image is input to the downsampling layer 102 as an input item of the downsampling layer 102, and the first feature image is downsampled by the downsampling layer 102 to output a second feature image corresponding to the third image (the second feature image is a feature image of the third image generated by the encoder), wherein the image size of the second feature image is smaller than the image size of the third image. The decoder 2000 includes an upsampling layer 201 and a second redundant learning layer 202 arranged in sequence, the feature image output by the encoder 1000 is input to the upsampling layer 201, and a third feature image is output after upsampling by the upsampling layer 201, the third feature image is input to the second redundant learning layer 202, and a generated image is output after passing through the second redundant learning layer 202, wherein the image size of the generated image is the same as the image size of the third image. This embodiment adopts an encoder-decoder structure to perform multi-scale training on the second preset network model, thereby improving the de-ghosting effect of the trained second image processing model.

[0222] Further, if Fig.19As shown, the first redundant learning layer 101 includes a first convolution layer 11 and a first redundant learning module 12, the downsampling layer 102 includes a first coding redundant learning module 110 and a second coding redundant learning module 120, the first coding redundant learning module 110 includes a first downsampling convolution layer 13 and a second redundant learning module 14, and the second coding redundant learning module 120 includes a second downsampling convolution layer 15 and a third redundant learning module 16. The input item of the first convolution layer 11 is a third image, and the third image is sampled to obtain a first feature image, and the first feature image is input to the first redundant learning module 12 for feature extraction, and the first feature image after the first redundant learning module 12 is downsampled in turn through the first downsampling convolution layer, the second redundant learning module 14, the second downsampling convolution layer 15 and the third redundant learning module 16 to obtain a second feature image. It can be seen that the first convolution layer 11 samples the third image, the first down-sampling convolution layer 13 and the second down-sampling convolution layer 15 are both used to down-sample the feature image input thereto, and the first redundant learning module 12, the second redundant learning module 14 and the third redundant learning module 16 are used to extract image features. In addition, in a possible implementation of this embodiment, the first down-sampling convolution layer 13 and the second down-sampling convolution layer 15 can both be convolution layers with a step size of 2, and the first redundant learning module 12, the second redundant learning module 14 and the third redundant learning module 16 each include three redundant learning blocks arranged in sequence, and the three redundant learning blocks extract image features of the input image in sequence.

[0223] For example: assuming that the third image is a 256*256 image, the third image is input into the first redundant learning layer 101 through the input layer, and after passing through the first redundant learning layer 101, a 256*256 first feature image is output; the first feature image is input into the first down-sampling convolution layer 13 of the first coding redundant learning module 110, and after being processed by the first down-sampling convolution layer 13, the image size is a fourth feature image of 128*128, and the fourth feature image is subjected to feature extraction by the first redundant learning module 14 of the first coding redundant learning module 110; the fourth feature image after passing through the first redundant learning module 110 is input into the second down-sampling convolution layer 15 of the second coding redundant learning module 120, and after being processed by the second down-sampling convolution layer 15, the image size is a second feature image of 64*64, and the second feature image is subjected to feature extraction by the second redundant learning module 16 of the second coding redundant learning module 120.

[0224] Further, if Fig.19As shown, the upsampling layer 201 includes a first decoding redundant learning module 210 and a second decoding redundant learning module 220, the first decoding redundant learning module 210 includes a fourth redundant learning module 21 and a first upsampling convolutional layer 22, the second decoding redundant learning module 220 includes a fifth redundant learning module 23 and a second upsampling convolutional layer 24, and the second redundant learning layer 202 includes a sixth redundant learning module 25 and a second convolutional layer 26. The input item of the first upsampling convolutional layer 22 is a first feature image, and the input first feature image is sequentially upsampled through the fourth redundant learning module 21, the first upsampling convolutional layer 22, the fifth redundant learning module 23 and the second upsampling convolutional layer 24 to obtain a third feature image, and the third feature image is input to the sixth redundant learning module 25, and the third feature image after feature extraction by the sixth redundant learning module 25 is input to the second convolutional layer 26, and the generated image is obtained through the second convolutional layer 26. It can be seen that the first upsampling convolution layer 22 and the second upsampling convolution layer 24 are used to upsample the feature image input thereto, the fourth redundant learning module 21, the fifth redundant learning module 23 and the sixth redundant learning module 25 are all used to extract image features, and the second convolution layer 26 is used to sample the feature image input therein. In a possible implementation of this embodiment, the first upsampling convolution layer 22 and the second upsampling convolution layer 24 are both deconvolution layers with a step size of 2, the fourth redundant learning module 21, the fifth redundant learning module 23 and the sixth redundant learning module 25 each include three redundant learning blocks, and the three redundant learning blocks sequentially extract the image features of the input image. In addition, the third redundant learning block of the redundant learning module in the first redundant learning layer 101 is jump-connected to the first redundant learning block of the redundant learning module in the second redundant learning layer 202, and the third redundant learning block of the redundant learning module in the first encoding redundant learning module 110 is jump-connected to the first redundant learning block of the redundant learning module in the second decoding redundant learning module 220.

[0225] For example: assuming that the third image is a 256*256 image, the encoder 1000 is used to obtain a 64*64 second feature image, the 64*64 second feature image is input to the fourth redundant learning module 21 of the first decoding redundant learning module 210 for feature extraction, the 64*64 second feature image after feature extraction is input to the first upsampling layer 22 of the first decoding redundant learning module 210, the image size of the fifth feature image output by the first upsampling layer 22 is 128*128, the fifth feature image is subjected to feature extraction by the fifth redundant learning module 23 of the second decoding redundant learning module 220; the fifth feature image after the fifth redundant learning module 23 is output to the second upsampling convolution layer 24 of the second decoding redundant learning module 220, the image size of the third feature image after the second upsampling convolution layer 24 is 256*256, the third feature image is input to the second redundant learning layer 202, and the generated image of 256*256 is output after the second redundant learning layer 202.

[0226] Furthermore, the first convolution layer, the second convolution layer, the first upsampling convolution layer, the second upsampling convolution layer, the first down-convolution layer and the second down-convolution layer included in the encoder and the decoder all use a linear rectification function as an activation function and the convolution kernels are all 5*5, which can improve the gradient transfer efficiency of each layer, and after multiple back propagation, the gradient amplitude changes little, thereby improving the accuracy of the trained generator and increasing the receptive field of the network.

[0227] L200. The second preset network model modifies the model parameters according to the fourth image corresponding to the third image and the generated image corresponding to the third image, and continues to execute the step of generating the generated image corresponding to the third image according to the third image in the next training image group in the training image set, until the training status of the second preset network model meets the preset conditions to obtain the second image processing model.

[0228] Specifically, the correction of the second preset network model refers to correcting the model parameters of the second preset network model until the model parameters meet the preset conditions. The preset conditions include that the loss function value meets the preset requirements or the number of training times reaches the preset number of times. The preset requirements can be determined according to the accuracy of the second image processing model, which is not described in detail here. The preset number of times can be the maximum number of training times of the second preset network model, for example, 4000 times. Therefore, the second preset network model outputs a generated image, and the loss function value of the second preset network model is calculated according to the generated image and the fourth image. After the loss function value is calculated, it is determined whether the loss function value meets the preset requirements; if the loss function value meets the preset requirements, the training is terminated; if the loss function value does not meet the preset requirements, it is determined whether the number of training times of the second preset network model reaches the prediction number, and if it does not reach the preset number of times, the network parameters of the second preset network model are corrected according to the loss function value; if it reaches the preset number of times, the training is terminated. In this way, judging whether the training of the second preset network model is terminated by the loss function value and the number of training times can avoid the training of the second preset network model entering an infinite loop due to the failure of the loss function value to meet the preset requirements.

[0229] Furthermore, since the network parameters of the second preset network model are modified when the training situation of the second preset network model does not meet the preset conditions (for example, the loss function value does not meet the preset requirements and the number of training times does not reach the preset number of times), after the network parameters of the second preset network model are corrected according to the loss function value, it is necessary to continue to train the network model, that is, continue to execute the step of inputting the third image in the training image set into the second preset network model. Among them, the third image that continues to be input into the second preset network model from the third image in the training image set can be a third image that has not been input into the second preset network model as an input item. For example, all third images in the training image set have a unique image identifier (for example, an image number), and the image identifier of the third image input into the second preset network model for the first training is different from the image identifier of the third image input into the second preset network model for the second training, such as the image number of the third image input into the second preset network model for the first training is 1, the image number of the third image input into the second preset network model for the second training is 2, and the image number of the third image input into the second preset network model for the Nth training is N. Of course, in practical applications, since the number of third images in the training image set is limited, in order to improve the training effect of the second image processing model, the third images in the training image set can be sequentially input into the second preset network model to train the second preset network model. After all the third images in the training image set have been input into the second preset network model, the operation of sequentially inputting the third images in the training image set into the second preset network model can be continued, so that the training image group in the training image set is cyclically input into the second preset network model. It should be noted that in the process of inputting the third images into the second preset network model for training, the images can be input in the order of the image numbers of the third images, or not in the order of the image numbers of the third images. Of course, the same third image can be repeatedly used to train the second preset network model, or the same third image can not be repeatedly used to train the second preset network model. In this embodiment, the specific implementation method of "continuing to execute the step of inputting the third images in the training image set into the second preset network model" is not limited.

[0230] Furthermore, in an implementation of this embodiment, the loss function value is calculated by using a structural similarity loss function and a content bidirectional loss function. Fig. 20 As shown, the second preset network model modifies the model parameters of the second preset network model according to the fourth image corresponding to the third image and the generated image corresponding to the third image, and continues to perform the step of generating the generated image corresponding to the third image according to the third image in the training image set until the training condition of the second preset network model meets the preset conditions to obtain the trained second image processing model, which specifically includes:

[0231] L201. Calculate a structural similarity loss function value and a content bidirectional loss function value corresponding to the second preset network model according to a fourth image corresponding to the third image and a generated image corresponding to the third image;

[0232] L202. Obtaining a total loss function value of the second preset network model according to the structural similarity loss function value and the content bidirectional loss function value;

[0233] L203. Train the second preset network model based on the total loss function value, and continue to execute the step of inputting the third image in the next training image group in the second preset training image set into the second preset network model until the training status of the second preset network model meets the preset conditions to obtain a trained second image processing model.

[0234] Specifically, the second preset network model adopts a combination of a structural similarity index (SSIM) loss function and a contextual bilateral loss (CoBi) loss function based on VGG (Visual Geometry Group Network, VGG network) feature extraction as a loss function. Then, when calculating the loss function value of the second preset network model, the structural similarity loss function value and the content bilateral loss function value can be calculated respectively, and then the loss function value in the second preset network model is calculated according to the structural similarity loss function value and the content bilateral loss function value. In this embodiment, the total loss function value of the second preset network model = a*structural similarity loss function value + b*content bilateral loss function value, where a and b are weight coefficients. For example, if the weight coefficient a and the weight coefficient b are both 1, then the total loss function value of the second preset network model = structural similarity loss function value + content bilateral loss function value. In addition, in this embodiment, when the total loss function value is used to train the second preset network model, the stochastic gradient descent method is used to train the second preset network model, wherein the initial network parameters of the training are set to 0.0001, and the network parameters are corrected in an exponential decay manner.

[0235] Furthermore, the structural similarity loss function value is used to measure the structural similarity between the generated image and the fourth image. The larger the structural similarity loss function value is, the higher the similarity between the generated image and the fourth image is. Conversely, the smaller the structural similarity loss function value is, the lower the similarity between the generated image and the fourth image is. Therefore, the structural similarity loss function value is more sensitive to local structural changes and is closer to the perception system of the human eye, thereby improving the accuracy of the second preset network model. In this embodiment, the expression of the structural similarity loss function corresponding to the structural similarity loss function value can be:

[0236]

[0237] Among them, μ x is the average value of all pixels in the generated image, μ y is the average value of all pixel values ​​in the second image, σ x is the variance of the pixel values ​​of all pixels in the generated image, σ y is the variance of the pixel values ​​of all pixels in the second image, σ xy is the covariance between the generated image and the second image.

[0238] Further, the content bidirectional loss function value is calculated by the CoBi loss function based on the VGG feature, and the CoBi loss function based on the VGG feature extracts several groups of VGG features of the generated image and the fourth image respectively, and for each first VGG feature of the generated image, searches for a second VGG feature close to the first VGG feature in the second VGG feature of the fourth image, and finally calculates the sum of the distances between each first VGG feature and its matching second VGG feature to obtain the content bidirectional loss function value. In this way, the bilateral distance is searched by the content bidirectional loss function, and the spatial loss of the first VGG feature and its matching second VGG feature is taken into account, so that the influence caused by the incomplete alignment of the third image and the fourth image can be avoided, and the speed and accuracy of the second preset network model training are improved. In addition, when searching for the second VGG feature matching the first VGG feature, the content bidirectional loss function value is determined according to the distance and position relationship between the first VGG feature and the second VGG feature, which improves the accuracy of the matching, thereby further reducing the influence of the misalignment of the third image and the fourth image on the training of the second preset network model. In this embodiment, the expression of the content bidirectional loss function can be:

[0239]

[0240] Where D is the cosine distance between the VVG feature of the generated image and the VVG feature of the second image, D′ is the spatial position distance between the VVG feature of the generated image and the VVG feature of the second image, N is the number of features of the VVG feature of the generated image, ω s is the weight coefficient.

[0241] N200. Perform deghosting processing on the processed image using the second image processing model, and use the deghosting processed image as the processed image.

[0242] Specifically, the deghosting of the image to be processed by the second image processing model refers to inputting the image to be processed into the second image processing model as an input item of the second image processing model, and removing the ghost of the image to be processed by the second image processing model to obtain a processed image, wherein the processed image is an image obtained by processing the second image processing model corresponding to the image to be processed. It can be understood that the image to be processed is an image with ghosting corresponding to the output image, that is, the output image corresponds to the image to be processed, they present the same image scene, the output image is a normally displayed image, the image content of the image to be processed corresponds to the output image, but the objects in the content of the image to be processed appear ghosting or a blurring effect similar to ghosting. For example, Fig.21 The image to be processed shown in FIG. 1 is obtained by performing the image processing as described above. Fig. 22 The output image is shown.

[0243] Further, it can be known from the training process of the second image processing model that the second image processing model includes an encoder and a decoder, so when the image to be processed is processed by the second image processing model, it needs to be processed by the encoder and the decoder respectively. Accordingly, the deghosting of the image to be processed by the second image processing model to obtain a processed image corresponding to the image to be processed specifically includes:

[0244] N201, inputting the image to be processed into the encoder, and obtaining a feature image of the image to be processed through the encoder, wherein the image size of the feature image is smaller than the image size of the image to be processed;

[0245] N202. Input the feature image into the decoder, and output a processed image corresponding to the image to be processed through the decoder, wherein the image size of the processed image is equal to the image size of the image to be processed.

[0246] Specifically, the encoder converts the input image to be processed into a feature image whose image space size is smaller than the input image and whose number of channels is greater than the input image, and inputs the feature image into the decoder, and the decoder converts the input feature image into a generated image with the same image size as the image to be processed. Among them, the structure of the encoder is the same as that of the encoder in the second preset network model, and the specific description of the structure of the encoder in the second preset network model can be referred to. The processing of the image to be processed by the encoder of the second image processing model is the same as the processing process of the third image by the encoder in the second preset network model, so the specific execution process of the step N201 can refer to step L101. Similarly, the structure of the decoder is the same as that of the decoder in the second preset network model, and the specific description of the structure of the decoder in the second preset network model can be referred to. The processing of the feature image corresponding to the image to be processed by the decoder of the second image processing model is the same as the processing process of the feature image corresponding to the third image by the decoder in the second preset network model, so the specific execution process of the step N202 can refer to step L202.

[0247] It is understandable that the network structure corresponding to the second image processing model during the training process is the same as the network structure corresponding to the application process (removing the ghost carried by the processed image). For example, during the training process, the second image processing model includes an encoder and an encoder, then correspondingly, when removing the ghost carried by the processed image through the second image processing model, the second image processing model also includes an encoder and an encoder.

[0248] Further, for example, during the training process, the encoder of the second image processing model includes the encoder including a first redundant learning layer and a downsampling layer, and the decoder includes an upsampling layer and a second redundant learning layer; accordingly, when removing ghosts carried by the processed image through the second image processing model, the encoder may also include a first redundant learning layer and a downsampling layer, and the decoder includes an upsampling layer and a second redundant learning layer; and during the application process, the working principle of each layer is the same as the working principle of each layer during the training process. Therefore, the input and output conditions of each layer of the neural network during the application process of the second image processing model can be referred to the relevant introduction during the training process of the second image processing model, which will not be repeated here.

[0249] In order to further improve the image quality of the output image obtained by the image processing method, after the output image is obtained, the output image may be post-processed, wherein the post-processing may include sharpening processing and noise reduction processing. Accordingly, after the processed image is deghosted by the second image processing model to obtain the output image, the method further includes:

[0250] The processed image is sharpened and denoised, and the output image after the sharpening and denoising is used as the output image corresponding to the image to be processed.

[0251] Specifically, the sharpening process refers to compensating the contour of the output image, enhancing the edge of the output image and the grayscale jump part to improve the image quality of the processed image. The sharpening process can adopt the existing sharpening process method, such as high-pass filtering method. The denoising process refers to removing the noise in the image and improving the signal-to-noise ratio of the image. The denoising process can adopt the existing denoising algorithm or the trained denoising network model, for example, the denoising process adopts the Gaussian low-pass filtering method.

[0252] Based on the above-mentioned image processing method, this embodiment provides a computer-readable storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the steps in the image processing method as described in the above-mentioned embodiment.

[0253] Based on the above image processing method, the present invention also provides a terminal device, such as Fig.23 As shown, it includes at least one processor 31; a display panel 31; and a memory 32, and may also include a communications interface 33 and a bus 34. The processor 30, the display panel 31, the memory 32 and the communications interface 33 may communicate with each other through the bus 34. The display panel 31 is configured to display a preset user guide interface in the initial setting mode. The communications interface 33 may transmit information. The processor 30 may call the logic instructions in the memory 32 to execute the method in the above embodiment.

[0254] In addition, the logic instructions in the memory 32 described above can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product.

[0255] The memory 32 is a computer-readable storage medium that can be configured to store software programs, computer executable programs, such as program instructions or modules corresponding to the methods in the embodiments of the present disclosure. The processor 30 executes functional applications and data processing by running the software programs, instructions or modules stored in the memory 32, that is, implementing the methods in the above embodiments.

[0256] The memory 32 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory 32 may include a high-speed random access memory and may also include a non-volatile memory. For example, a variety of media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, may also be a transient storage medium.

[0257] In addition, the specific process of loading and executing the plurality of instructions in the storage medium and the processor in the terminal has been described in detail in the method above, and will not be described here one by one.

[0258] In addition, the specific process of loading and executing the multiple instruction processors in the above storage medium and the terminal device has been described in detail in the above method and will not be described here one by one.

[0259] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire a set of images to be processed, wherein the set of images to be processed includes a plurality of images; According to the set of images to be processed, generating a denoised image corresponding to the set of images to be processed; Inputting the denoised image into a first image processing model that has been trained, performing a decolorization process on the denoised image through the image processing model, and obtaining a processed image corresponding to the denoised image; Inputting the processed image into a trained second image processing model, and performing a deghosting process on the processed image by the second image processing model to obtain an output image; Among the multiple images included in the image set to be processed, one image is a basic image, and the remaining images are adjacent images of the basic image. The step of generating a denoised image corresponding to the image set to be processed according to the image set to be processed specifically includes: Dividing the basic image into a plurality of basic image blocks, and respectively determining adjacent image blocks corresponding to each basic image in each adjacent image; Determine the weight parameter sets corresponding to the basic image blocks respectively; wherein the weight parameter sets corresponding to the basic image blocks include a first weight parameter and a second weight parameter, the first weight parameter is a weight parameter of the basic image block, and the second weight parameter is a weight parameter of an adjacent image block in an adjacent image corresponding to the basic image block; A denoised image is determined according to the set of images to be processed and the weight parameter sets respectively corresponding to the basic image blocks.

2. The image processing method according to claim 1, characterized in that: The number of images in the to-be-processed image set is determined according to shooting parameters corresponding to the to-be-processed image set.

3. The image processing method according to claim 1, characterized in that: The image definition of the basic image is greater than or equal to the image definition of the adjacent image.

4. The image processing method according to claim 1, characterized in that: The step of determining the weight parameter sets corresponding to the basic image blocks specifically includes: For each basic image block, second weight parameters of adjacent image blocks corresponding to the basic image block are determined, and first weight parameters corresponding to the basic image block are acquired to obtain a weight parameter set corresponding to the basic image block.

5. The image processing method according to claim 4, characterized in that: The step of determining the second weight parameters of each adjacent image block corresponding to the basic image block specifically includes: For each adjacent image block, calculating a similarity value between the basic image block and the adjacent image block; A second weight parameter of the adjacent image block is calculated according to the similarity value.

6. The image processing method according to claim 5, characterized in that: Calculating the second weight parameter of the adjacent image block according to the similarity value specifically includes: When the similarity value is less than or equal to a first threshold, using the first preset parameter as a second weight parameter of the adjacent image block; When the similarity value is greater than a first threshold value and less than or equal to a second threshold value, calculating a second weight parameter of the adjacent image block according to the similarity value, the first threshold value and the second threshold value; When the similarity value is greater than a second threshold, a second preset parameter is preset as a second weight parameter of the adjacent image block.

7. The image processing method according to claim 6, characterized in that: The first threshold and the second threshold are both determined according to the similarity value between the basic image block and the adjacent image block.

8. The image processing method according to claim 1, characterized in that: After determining the denoised image according to the image set to be processed and the weight parameter sets respectively corresponding to the basic image blocks, the method further comprises: The denoised image is subjected to spatial denoising, and the image obtained after the spatial denoising is used as the denoised image.

9. The image processing method according to claim 1, characterized in that: The first image processing model is trained based on a first training image set, the first training image set includes multiple training image groups, each training image group includes a first image and a second image, and the first image is a color cast image corresponding to the second image.

10. The image processing method according to claim 9, characterized in that: The first image is an image captured by an under-screen imaging system.

11. The image processing method according to claim 9, characterized in that: Before training the first image processing model based on the first training image set, the following steps are included: For each training image group in the first training sample set, align the first image in the training image group with the second image corresponding to the first image to obtain an aligned image aligned with the second image, and use the aligned image as the first image.

12. The image processing method according to claim 9, characterized in that: The first training image set includes several training sub-image sets, each training sub-image set includes several groups of training sample image groups, the exposure of the first images in any two groups of training sample image groups in the several groups of training image groups is the same, the exposure of the second image in each group of training sample image groups in the several groups of training image groups is within a preset range, and the exposure of the first images in any two training sub-image sets are different.

13. The image processing method according to claim 12, characterized in that: The first image processing model corresponds to a plurality of model parameters, each of which is obtained by training based on a training sub-image set in the first training image set, and the training sub-image sets corresponding to any two model parameters are different.

14. The image processing method according to claim 13, characterized in that: The step of inputting the denoised image into the trained first image processing model specifically comprises: Extracting the exposure of the denoised image; Determining model parameters corresponding to the denoised image according to the exposure, and updating model parameters of the first image processing model using the model parameters; The denoised image is input into the updated first image processing model.

15. The image processing method according to claim 14, characterized in that: The step of extracting the exposure of the denoised image specifically includes: Determine a third pixel point that meets a preset condition according to the R value, the G value, and the B value of each pixel point in the denoised image, wherein the preset condition is that at least one of the R value, the G value, and the B value is greater than a preset threshold; A highlight area of ​​the denoised image is determined according to all third pixels that meet a preset condition, and an exposure of the denoised image is determined according to the highlight area.

16. The image processing method according to claim 15, characterized in that: Determining the highlight area of ​​the denoised image according to all third pixel points that meet the preset condition specifically includes: Obtaining a connected area formed by all third pixel points that meet the preset condition, and selecting a target area that meets a preset rule from all the obtained connected areas, wherein the preset rule is that the types of the R values, G values, and / or B values ​​of the third pixel points in the target area that are greater than a preset threshold are the same; The areas corresponding to the selected target areas are calculated, and the target area with the largest area is selected as the highlight area.

17. The image processing method according to claim 16, characterized in that: The step of determining the exposure of the denoised image according to the highlight area specifically includes: Calculating a first area of ​​the highlight region and a second area of ​​the denoised image; The exposure degree corresponding to the denoised image is determined according to the ratio of the first area to the second area.

18. The image processing method according to any one of claims 1 to 17, characterized in that: The first image processing model includes a downsampling module and a transformation module; the decolorization processing of the denoised image by the first image processing model to obtain a processed image corresponding to the denoised image specifically includes: Inputting the denoised image into the downsampling module, obtaining a bilateral grid corresponding to the denoised image and a guidance image corresponding to the denoised image through the downsampling module, wherein a resolution of the guidance image is the same as a resolution of the denoised image; The guiding image, the bilateral grid and the denoised image are input into the transformation module, and a processed image corresponding to the denoised image is generated by the transformation module.

19. The image processing method according to claim 18, characterized in that: The downsampling module includes a downsampling unit and a convolution unit; the step of inputting the denoised image into the downsampling module, and obtaining the bilateral grid corresponding to the denoised image and the guidance image corresponding to the denoised image through the downsampling module specifically includes: Inputting the denoised image into the downsampling unit and the convolution unit respectively; A bilateral grid corresponding to the denoised image is obtained through the downsampling unit, and a guidance image corresponding to the denoised image is obtained through the convolution unit.

20. The image processing method according to claim 19, characterized in that: The transformation module includes a segmentation unit and a transformation unit, and the step of inputting the guide image, the bilateral grid, and the denoised image into the transformation module, and generating a processed image corresponding to the denoised image through the transformation module specifically includes: Inputting the guidance image into the segmentation unit, segmenting the bilateral grid by the segmentation unit to obtain a color transformation matrix of each pixel in the denoised image; The denoised image and the color transformation matrix of each pixel in the denoised image are input into the transformation unit, and a processed image corresponding to the denoised image is generated by the transformation unit.

21. The image processing method according to claim 1, characterized in that: The second image processing model is trained based on a second training image set, the second training image set includes multiple training image groups, each training image group includes a third image and a fourth image, and the third image is an image with ghosting corresponding to the fourth image.

22. The image processing method according to claim 21, characterized in that: The third image is generated based on the fourth image and a point spread function, wherein the point spread function is generated based on a grayscale image generated by a shading structure in an under-screen imaging system.

23. The image processing method according to claim 21, characterized in that: The third image is an image captured by the under-screen imaging system.

24. The image processing method according to claim 23, characterized in that: The under-screen imaging system is an under-screen camera.

25. The image processing method according to claim 23, characterized in that: Before training the second image processing model based on the second training image set, the following step is also included: For each training image group in the second training image set, align the third image in the training image group with the fourth image corresponding to the third image to obtain an aligned image aligned with the fourth image, and use the aligned image as the third image.

26. The image processing method according to any one of claims 21 to 25, characterized in that: The second image processing model includes an encoder and a decoder; and the deghosting of the processed image by the second image processing model to obtain an output image corresponding to the processed image specifically includes: Inputting the processed image into the encoder, and obtaining a feature image of the processed image through the encoder, wherein the image size of the feature image is smaller than the image size of the processed image; The feature image is input into the decoder, and the decoder outputs an output image corresponding to the processed image, wherein the image size of the output image is equal to the image size of the processed image.

27. The image processing method according to claim 11 or 25, characterized in that: The alignment process specifically includes: Acquire a benchmark image and a reference image in a training image group, and calculate a pixel deviation between the reference image and the benchmark image; wherein, when the benchmark image is the second image, the reference image is the first image; and when the benchmark image is the fourth image, the reference image is the third image; An alignment method corresponding to the reference image is determined according to the pixel deviation amount, and the reference image is aligned with the benchmark image using the alignment method.

28. The image processing method according to claim 27, characterized in that: Determining the alignment method corresponding to the reference image according to the pixel deviation, and aligning the reference image with the benchmark image using the alignment method specifically includes: When the pixel deviation is less than or equal to a preset deviation threshold, aligning the reference image with the reference image as a reference according to the mutual information between the reference image and the reference image; When the pixel deviation is greater than the preset deviation threshold, extract a benchmark pixel point set of the reference image and a reference pixel point set of the benchmark image, wherein the benchmark pixel point set includes a number of reference pixel points in the reference image, and the benchmark pixel point set includes a number of benchmark pixel points in the benchmark image, and the reference pixel points in the reference pixel point set correspond one-to-one to the benchmark pixel points in the benchmark pixel point set; for each benchmark pixel point in the benchmark pixel point set, calculate the coordinate difference between the benchmark pixel point and the reference pixel point corresponding to it, and adjust the position of the reference pixel point according to the coordinate difference corresponding to the reference pixel point, so as to align the reference pixel point with the benchmark pixel point corresponding to the reference pixel point.

29. The image processing method according to any one of claims 1-17 or 21-25, characterized in that: After inputting the processed image into a trained second image processing model and performing a deghosting process on the processed image by the second image processing model to obtain an output image, the method further includes: The processed image is sharpened and denoised, and the processed image after the sharpening and denoising is used as the output image.

30. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the image processing method according to any one of claims 1 to 29.

31. A terminal, characterized in that: include: A processor and a memory; the memory stores a computer-readable program that can be executed by the processor; when the processor executes the computer-readable program, the steps in the image processing method according to any one of claims 1 to 30 are implemented.

Images