Photograph processing method, apparatus and device

Abstract

The application relates to a photo processing method, device and equipment. The photo processing method comprises the following steps: transforming an initial photo into an intermediate picture with unchanged resolution; performing sharpness optimization processing on a portrait area and a background area of the intermediate picture respectively; fusing the portrait area and the background area after the sharpness optimization processing to obtain a fused picture; and transforming the fused picture into a specified specification picture. The scheme provided by the application can improve the sharpness of a photo.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a photo processing method, device and equipment. BACKGROUND

[0002] In daily life, people often use ID photos. ID photos are photos used to prove identity, and generally have high requirements for the clarity of the photos. There are various ID photo generation and processing methods in the related art, which generally scale and transform the photos to a specified size and then perform related optimization processing.

[0003] However, the photo processing method in the related art inevitably causes a decrease in the clarity of the picture when scaling and transforming the photo to a specified size and then performing optimization processing, thereby affecting the clarity of the photo. SUMMARY

[0004] To solve or partially solve the problems in the related art, the present application provides a photo processing method, device and equipment, which can improve the clarity of the photo.

[0005] The first aspect of the present application provides a photo processing method, comprising:

[0006] transforming an initial photo into an intermediate picture with unchanged resolution;

[0007] performing clarity optimization processing on the intermediate picture according to a portrait region and a background region, respectively;

[0008] fusing the portrait region and the background region after the clarity optimization processing, respectively, to obtain a fused picture;

[0009] transforming the fused picture into a specified size picture.

[0010] In an embodiment, the step of transforming the initial photo into the intermediate picture with unchanged resolution comprises:

[0011] transforming the initial photo into the intermediate picture with unchanged resolution through a first affine transformation matrix.

[0012] In an embodiment, the first affine transformation matrix is obtained in the following manner:

[0013] aligning the initial photo to a reference picture to obtain a transformation parameter of a position coordinate;

[0014] constructing a transformation matrix according to the transformation parameter, and performing setting processing on the transformation matrix to obtain the first affine transformation matrix.

[0015] In an embodiment, after the initial photo is transformed into the intermediate picture with the first affine transformation matrix, if the picture resolution is greater than or equal to a set threshold, a first interpolation method is used for interpolation, and if the picture resolution is less than the set threshold, a second interpolation method is used for interpolation.

[0016] In an embodiment, the resolution optimization processing of the intermediate picture according to the portrait region and the background region respectively includes:

[0017] The resolution of the portrait region of the intermediate picture is optimized according to a first optimization algorithm;

[0018] The resolution of the background region of the intermediate picture is optimized according to a second optimization algorithm.

[0019] In an embodiment, the transformation of the fusion picture into the specified specification picture includes:

[0020] The fusion picture is transformed into the specified specification picture by a second affine transformation matrix;

[0021] The second affine transformation matrix is obtained in the following manner: the intermediate picture is aligned to the specified specification picture to obtain a transformation parameter of position coordinates; a transformation matrix is constructed according to the transformation parameter, and the transformation matrix is used as the second affine transformation matrix.

[0022] The second aspect of the present application provides a photo processing device, which includes:

[0023] A first transformation module is configured to transform an initial photo into an intermediate picture with invariant resolution;

[0024] An optimization processing module is configured to perform resolution optimization processing on the intermediate picture transformed by the first transformation module according to a portrait region and a background region respectively;

[0025] A fusion processing module is configured to fuse the portrait region and the background region after the resolution optimization processing by the optimization processing module, to obtain a fusion picture;

[0026] A second transformation module is configured to transform the fusion picture obtained by the fusion processing module into a specified specification picture.

[0027] In an embodiment, the first transformation module transforms the initial photo into the intermediate picture with invariant resolution by a first affine transformation matrix; or,

[0028] The second transformation module transforms the fusion picture into the specified specification picture by a second affine transformation matrix.

[0029] The third aspect of the present application provides an electronic device, which includes:

[0030] Processor; and

[0031] A memory that stores executable code, which, when executed by the processor, causes the processor to perform the method described above.

[0032] A fourth aspect of this application provides a computer-readable storage medium having executable code stored thereon, which, when executed by a processor of an electronic device, causes the processor to perform the method described above.

[0033] The technical solution provided in this application may include the following beneficial effects:

[0034] Related technologies first scale the initial photo before processing, but this scaling leads to a loss of sharpness during the photo processing workflow. The technical solution of this application, however, transforms the initial photo into an intermediate image with the same resolution, thus reducing or avoiding sharpness loss during processing. Furthermore, this application performs sharpness optimization processing on the intermediate image separately for the portrait area and the background area, and then merges the separately optimized portrait and background areas to obtain a merged image. This allows for different sharpness optimization processes based on the image characteristics of different areas within the image, further reducing or avoiding sharpness loss during processing. Therefore, the technical solution of this application can improve the sharpness of photos through optimization processing.

[0035] Furthermore, the technical solution of this application can transform the initial photo into an intermediate image with unchanged resolution using a first affine transformation matrix, and transform the fused image into an image of specified specifications using a second affine transformation matrix. By using two affine transformations, the loss of sharpness during the photo processing can also be reduced.

[0036] Furthermore, the technical solution of this application can optimize the clarity of the portrait area of ​​the intermediate image using a first optimization algorithm; and optimize the clarity of the background area of ​​the intermediate image using a second optimization algorithm. Because the portrait area is optimized separately, its clarity can be improved; and other background areas besides the portrait area are also optimized separately using other more suitable algorithms, thus improving their clarity as well. In this way, the overall clarity of the photograph is higher than that obtained using related processing techniques.

[0037] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0038] The above and other objects, features and advantages of the present application will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings in which like reference characters refer to like parts throughout the figures, and in which:

[0039] Figure 1 is a flowchart of a photo processing method according to an embodiment of the present application;

[0040] Figure 2 is another flowchart of a photo processing method according to an embodiment of the present application;

[0041] Figure 3 is a process diagram for determining an affine transformation matrix according to an embodiment of the present application; Figure 2 is a corresponding method application diagram according to an embodiment of the present application;

[0042] Figure 4 is a process diagram for determining an affine transformation matrix according to an embodiment of the present application;

[0043] Figure 5 is a structural diagram of a photo processing apparatus according to an embodiment of the present application;

[0044] Figure 6 is another structural diagram of a photo processing apparatus according to an embodiment of the present application;

[0045] Figure 7 is a structural diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0046] Embodiments of the present application will be described more fully hereinafter with reference to the accompanying drawings, in which embodiments of the application are shown. This application may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the application to those skilled in the art.

[0047] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0048] It should be understood that, although the terms "first", "second", "third", etc. can be used in this application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information without departing from the scope of the present application. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "a plurality of" is two or more, unless otherwise explicitly and specifically limited.

[0049] The photo processing method in the related art can cause the picture definition to decrease, thereby affecting the definition of the photo. In view of the above problem, the present application provides a photo processing method, which can improve the definition of the photo.

[0050] The technical solutions of the embodiments of the present application are described in detail below with reference to the drawings.

[0051] Figure 1 is a flowchart of the photo processing method shown in the embodiments of the present application.

[0052] Referring to Figure 1 , the method comprises:

[0053] In S101, the initial photo is transformed into an intermediate picture with unchanged resolution.

[0054] The step S101 can transform the initial photo into an intermediate picture with unchanged resolution through a first affine transformation matrix. The initial photo can be, for example, an ID photo, but is not limited thereto. The first affine transformation matrix can be obtained in the following manner: obtaining a transformation parameter of position coordinates by aligning the initial photo to a reference picture; constructing a transformation matrix according to the transformation parameter, and performing setting processing on the transformation matrix to obtain the first affine transformation matrix. The translation transformation matrix T, the rotation transformation matrix R and the scaling transformation matrix S can be constructed according to the transformation parameter; the translation transformation matrix T, the rotation transformation matrix R and the scaling transformation matrix S are subjected to left multiplication operation, and then the transformation matrix is obtained; the transformation matrix is subjected to decomposition operation, the scaling coefficient is modified to 1, and then the first affine transformation matrix is obtained. Since the scaling coefficient of the first affine transformation matrix has been modified to 1, the resolution of the photo after the step of transformation is unchanged, which can reduce the loss of definition of the picture.

[0055] In S102, the intermediate picture is subjected to definition optimization processing according to the portrait region and the background region respectively.

[0056] The step S102 can perform sharpness optimization processing on the portrait region of the intermediate picture according to a first optimization algorithm, and perform sharpness optimization processing on the background region of the intermediate picture according to a second optimization algorithm. The first optimization algorithm can be a face blind restoration algorithm, and the second optimization algorithm can be a picture super-resolution algorithm. Different optimization processing modes are used according to different characteristics of different regions of the picture, which can further improve the sharpness of the picture.

[0057] In S103, the portrait region and the background region which have respectively undergone sharpness optimization processing are fused to obtain a fused picture.

[0058] The step S103 can fuse the portrait region and the background region which have respectively undergone sharpness optimization processing by using a Poisson fusion algorithm to obtain a fused picture.

[0059] In S104, the fused picture is transformed into a picture of a specified specification.

[0060] The step S104 can transform the fused picture into a picture of a specified specification by using a second affine transformation matrix. The second affine transformation matrix can be obtained in the following manner: transforming the intermediate picture to a picture of a specified specification to obtain transformation parameters of position coordinates; constructing a transformation matrix according to the transformation parameters, and taking the transformation matrix as the second affine transformation matrix. A translation transformation matrix T, a rotation transformation matrix R, and a scaling transformation matrix S can be constructed according to the transformation parameters; and the translation transformation matrix T, the rotation transformation matrix R, and the scaling transformation matrix S are subjected to left multiplication operation, and then a transformation matrix is obtained, which is taken as the second affine transformation matrix.

[0061] As can be seen from the embodiment, the related art is to first perform scaling processing on the initial picture, and the scaling processing can cause loss of sharpness in the picture processing procedure. However, the technical solution of the present application transforms the initial picture into an intermediate picture with unchanged resolution, which can reduce or avoid loss of sharpness in the picture processing procedure. In addition, the present application performs sharpness optimization processing on the portrait region and the background region of the intermediate picture respectively, and then fuses the portrait region and the background region which have respectively undergone sharpness optimization processing to obtain a fused picture. In this way, different sharpness optimization processing can be performed according to the image characteristics of different regions of the picture, which can further reduce or avoid loss of sharpness in the picture processing procedure. Therefore, the technical solution of the embodiment of the present application can improve the sharpness of the picture through optimization processing.

[0062] Figure 2 is another flowchart of the picture processing method shown in the embodiment of the present application.

[0063] The technical scheme of the embodiment of the present application first transforms the initial photo to an intermediate picture through a first affine transformation matrix. Since the first affine transformation matrix has modified the scaling coefficient to 1, the resolution of the photo after the transformation is not changed (compared with the original photo, the same area corresponding to the original photo does not change in resolution, the whole picture involves rotation and cropping, and the size may change to a certain extent), only the rotation and translation correction is performed; then the intermediate picture is respectively processed for sharpness optimization according to the portrait area and the background area and a fusion picture is obtained, and finally the fusion picture is transformed to a specified size through a second affine transformation matrix. Through the above processing, the sharpness loss problem in the processing process is reduced, the effect of optimizing sharpness is improved, and the overall sharpness of the photo is better optimized.

[0064] Referring to Figure 2 and Figure 3 , the method comprises:

[0065] In S201, the initial photo is transformed into an intermediate picture with unchanged resolution through a first affine transformation matrix.

[0066] Taking the initial photo as a portrait picture and the reference picture as a front face reference picture as an example, the determination process of the first affine transformation matrix is introduced, which comprises: Figure 4 ) including:

[0067] 1) According to the position coordinates of the face key points of the portrait picture and the face key points of the front face reference picture, the transformation matrix tfm is obtained.

[0068] 1-1) Align the initial photo to the reference picture to obtain the transformation parameters of the position coordinates.

[0069] Suppose the position coordinates of the face key points of the portrait picture are (x, y), and the position coordinates of the face key points of the front face reference picture are (x', y'), after aligning the face key points of the portrait picture to the face key points of the front face reference picture, the transformation parameters of the position coordinates of the face key points of the portrait picture to the position coordinates of the face key points of the front face reference picture can be calculated. The said alignment includes translation, rotation and scaling processing of the portrait picture. The transformation parameters include translation parameters tx and ty for translation processing, rotation parameters θ for rotation processing, and scaling coefficients sx and sy for scaling processing.

[0070] Among them, the face key points can be selected from a set number of key feature points of the face, for example, 3 or more key feature points, for example, 5 key feature points of left eye, right eye, nose, left corner of mouth and right corner of mouth, but not limited to this.

[0071] 1-2) Construct a translation transformation matrix T, a rotation transformation matrix R and a scaling transformation matrix S according to the transformation parameters:

[0072]

[0073] Wherein T, R, S respectively represent translation transformation matrix, rotation transformation matrix and scaling transformation matrix, t represents the parameter of translation, s represents the scaling factor, θ represents the angle of rotation, and subscripts x and y represent the corresponding components of the coordinate position on the x-axis and y-axis.

[0074] Wherein, translation is to move each point to (x+t, y+t); rotation is to rotate clockwise around the origin by an angle θ, and scaling is to enlarge or reduce the horizontal coordinate of each point by sx times and the vertical coordinate by sy times.

[0075] 1-3) Multiplying the translation transformation matrix T, the rotation transformation matrix R and the scaling transformation matrix S, the transformation matrix tfm is obtained:

[0076]

[0077] Formula (2) represents that R transformation is performed first, then S transformation is performed, and then T transformation is performed. That is, the transformation order is to perform the transformation in the back of the formula first, and then perform the transformation in the front, and the matrix multiplication is left multiplication. The matrix left multiplication can be understood as multiplying the matrix on the left side of the multiplication sign.

[0078] The transformation relationship between the face key points of the frontal reference image and the face key points of the portrait image can be:

[0079]

[0080] It can be seen that through the alignment processing, the transformation matrix tfm of the portrait image aligned to the frontal reference image can be calculated.

[0081] 2) Decompose the transformation matrix tfm to modify the scaling factor to 1 to obtain the first affine transformation matrix tfm_stage1.

[0082] This step decomposes the transformation matrix tfm, and the decomposition process is to decompose tfm into S·tfm_stage1. In theory, the transformation matrix tfm can be decomposed into any multiple, for example, tfm=tfm_n·tfm_n-1·....·tfm_2·tfm_1. In the technical solution of the present application, the decomposition is to perform tfm_stage1 transformation first, and then perform S transformation, so that it is equivalent to the transformation matrix tfm. Because the S transformation is known, tfm_stage1 can be decomposed.

[0083] The decomposition operation of tfm is as follows:

[0084]

[0085] The calculation formula of the scaling coefficients sx and sy in the rotation matrix S can be as follows:

[0086]

[0087] Then, the scaling coefficients are modified to 1, i.e., S is changed to 1. tfm_stage1 is multiplied by 1, and then the diagonal matrix is equivalent to no multiplication. That is, the scaling coefficients are 1, which means that sx and sy in the rotation transformation matrix S are changed to 1, which is equivalent to no S transformation, and the remaining is tfm_stage1.

[0088] Therefore, the first affine transformation matrix required for transformation to the intermediate picture is as follows:

[0089]

[0090] 3) Transform the portrait picture into a resolution-invariant intermediate picture through the first affine transformation matrix tfm_stage1.

[0091] Since the scaling coefficients in the first affine transformation matrix tfm_stage1 are modified to 1, which is equivalent to no S transformation, the resolution of the transformed portrait picture is not changed, and finally the portrait picture is processed through the first affine transformation matrix tfm_stage1 for rotation and translation, and is transformed into a resolution-invariant intermediate picture.

[0092] After the initial picture is transformed into a resolution-invariant intermediate picture through the first affine transformation matrix (compared with the original picture of the initial picture, the resolution of the same region corresponding to the original picture is not changed, the whole picture involves rotation and cropping, and the size can change to a certain extent), if the picture resolution of the transformed picture (for example, after rotation and cropping) is greater than or equal to a set threshold, a first interpolation method is used for interpolation, and if the picture resolution is less than the set threshold, a second interpolation method is used for interpolation. The first interpolation method can be, for example, INTER_CUBIC interpolation (cubic spline interpolation); and the second interpolation method can be, for example, INTER_AREA (area interpolation) interpolation. For a picture with a larger resolution, the INTER_CUBIC interpolation method has better scaling effect, and for compressing a face to a picture with a smaller resolution, the INTER_AREA interpolation method can avoid the graininess caused by INTER_CUBIC.

[0093] The implementation of affine transformation using OpenCV (a cross-platform computer vision and machine learning software library based on Apache 2.0 license (open source)) generally involves the warpAffine function, which can implement some simple remapping. In the scheme of the present application, the use of the warpAffine function is modified. After the portrait picture is rotated and translated by the warpAffine function, the picture is scaled. For pictures with a larger resolution, for example, if the short side of the picture is greater than or equal to 260, INTER_CUBIC interpolation method is used for interpolation, and if the short side of the picture is less than 260, INTER_AREA interpolation method is used for interpolation.

[0094] Since the scaling coefficient of the first affine transformation matrix tfm_stage1 is modified to 1 in the present application, the modification of the warpAffine function does not affect the use of the first affine transformation matrix tfm_stage1, but improves the effect after the application of the second affine transformation matrix tfm_stage2.

[0095] In S202, the portrait area of the intermediate picture is processed for sharpness optimization according to a first optimization algorithm.

[0096] The face is a region of interest in the picture, also known as the portrait area. The face sharpness enhancement is performed separately in the embodiment scheme of the present application. In this step S202, the portrait area of the intermediate picture is processed for sharpness optimization according to a first optimization algorithm. The first optimization algorithm may, for example, be a face blind restoration algorithm.

[0097] Face blind restoration refers to the reconstruction of a degraded (low resolution, noise, blur, picture compression, etc.) face photo, ultimately obtaining a less degraded and clear face photo. Face blind restoration mainly repairs the face with the help of geometric face prior information (key points, face segmentation, face heat map). Among them, GFPGAN is the most advanced model of face blind restoration algorithm, which uses generated face prior to perform face blind restoration. The present application can use the GFPGAN model for face sharpness optimization but is not limited thereto.

[0098] In S203, the background area of the intermediate picture is processed for sharpness optimization according to a second optimization algorithm.

[0099] In the embodiment of the present application, because the face region is processed and optimized separately, this step is mainly to enhance the clarity of the background other than the face. The present application can perform clarity optimization processing on the background region of the intermediate picture according to a second optimization algorithm. The second optimization algorithm may, for example, be a picture super-resolution algorithm. The present application uses a super-resolution algorithm model, for example, a RealESRGAN model, to optimize the background clarity. The RealESRGAN model mainly simulates various “degradation” processes in the process of converting a high-resolution image into a low-resolution image, and then lets the model see a blurred picture and inversely deduces the high-definition picture.

[0100] The step S203 can be performed separately from the step S202, and there is no sequence relationship between the two steps.

[0101] It should be noted that, by using the picture super-resolution algorithm to convert the picture from low resolution to high resolution, the information contained in the picture is increased. If the information contained in the original picture is missing, the model after the clarity is reduced and the super-resolution processing is performed will also be affected. For example, the original picture is 1024x1024, and the picture is directly processed by the super-resolution algorithm without scaling according to the scheme of the present application to obtain a 2048x2048 picture A. The original picture is scaled to a 512x512 picture first, and then the scaled picture is processed by the super-resolution algorithm to obtain a 2048x2048 picture B. Because the picture B is scaled first, the clarity is reduced, and therefore the picture A obtained by the scheme of the present application has better clarity than the picture B obtained by the related art. The same is true for the face blind restoration algorithm.

[0102] It should also be noted that, if other portrait processing logic is required according to different business requirements, the intermediate picture can also be processed accordingly, and the embodiment of the present application is not limited in this regard.

[0103] In S204, the face region and the background region processed by the clarity optimization are fused by using the Poisson fusion algorithm to obtain a fused picture.

[0104] The present application applies the Poisson fusion algorithm to paste the face with enhanced clarity back to the picture with enhanced background clarity, completes the picture clarity enhancement, and obtains a processed fused picture.

[0105] Image fusion is a key technology in image stitching technology, and its principle is to realize smooth transition and seamless stitching between the stitched images by redefining and calculating the pixels in the overlapping area. The Poisson fusion algorithm based on Poisson equation uses the gradient field of two images to guide the difference value of the overlapping area, changes the image fusion problem into a minimum problem of solving the difference value between the gradient field of the target image block and the background guide gradient field, and can achieve good image fusion effect.

[0106] In S205, the fusion picture is transformed into a specified specification picture by a second affine transformation matrix.

[0107] This step determines the second affine transformation matrix tfm_stage2 of the intermediate picture aligned to the specified specification picture.

[0108] The principle of the determination process of the second affine transformation matrix tfm_stage2 and the first affine transformation matrix tfm_stage1 is the same (see Figure 4 ), and the transformation matrix tfm can be obtained according to the face key points of the intermediate picture and the face key points of the specified specification picture. However, at this time, the transformation matrix tfm does not need to be decomposed and operated, nor does it need to be modified to 1. Only the obtained transformation matrix tfm is directly used as the second affine transformation matrix tfm_stage2.

[0109] This step S205 transforms the fusion picture into a specified specification picture by a second affine transformation matrix, that is, applies an affine transformation to the fusion picture obtained after the fusion processing to obtain the final specified specification picture.

[0110] In summary of the above description, the technical scheme of the embodiment of the present application modifies the scaling factor of the affine transformation matrix to 1, so that after the initial picture is corrected by rotation and translation, the resolution of the picture does not change, and the interpolation method used in the scaling process of the affine transformation is specified, which can reduce the loss of clarity in the photo processing flow. Although the picture after the second affine transformation is finally used, the intermediate picture after the first affine transformation is used before the last step. The intermediate picture and the initial picture maintain the same resolution, thereby avoiding the loss of clarity in the processing process and improving the clarity of the processed picture.

[0111] Corresponding to the foregoing application function implementation method embodiment, the embodiment of the present application also provides a photo processing device, an electronic device and corresponding embodiments.

[0112] Figure 5 is a structural schematic diagram of the photo processing device shown in the embodiment of the present application.

[0113] See Figure 5A photo processing device 50 comprises a first transformation module 51, an optimization processing module 52, a fusion processing module 53, and a second transformation module 54.

[0114] The first transformation module 51 is configured to transform an initial photo into an intermediate picture with unchanged resolution. The first transformation module 51 transforms the initial photo into the intermediate picture with unchanged resolution by using a first affine transformation matrix. The first affine transformation matrix can be obtained by aligning the initial photo to a reference picture to obtain transformation parameters of position coordinates, and constructing a transformation matrix according to the transformation parameters, and performing setting processing on the transformation matrix to obtain the first affine transformation matrix. The transformation parameters can be used to construct a translation transformation matrix T, a rotation transformation matrix R, and a scaling transformation matrix S. The translation transformation matrix T, the rotation transformation matrix R, and the scaling transformation matrix S are subjected to left multiplication operation to obtain the transformation matrix. The transformation matrix is subjected to decomposition operation, and the scaling coefficient is modified to 1 to obtain the first affine transformation matrix.

[0115] The optimization processing module 52 is configured to perform sharpness optimization processing on the intermediate picture transformed by the first transformation module 51 according to a portrait region and a background region, respectively. The optimization processing module 52 can perform sharpness optimization processing on the portrait region of the intermediate picture according to a first optimization algorithm, and perform sharpness optimization processing on the background region of the intermediate picture according to a second optimization algorithm. The first optimization algorithm can be a face blind restoration algorithm, and the second optimization algorithm can be a picture super-resolution algorithm.

[0116] The fusion processing module 53 is configured to fuse the portrait region and the background region that are subjected to sharpness optimization processing by the optimization processing module 52, respectively, to obtain a fusion picture. The fusion processing module 53 can fuse the portrait region and the background region that are subjected to sharpness optimization processing, respectively, by using a Poisson fusion algorithm to obtain the fusion picture.

[0117] The second transformation module 54 is configured to transform the fusion picture obtained by the fusion processing module 53 into a specified specification picture. The second transformation module 54 can transform the fusion picture into the specified specification picture by using a second affine transformation matrix. The second affine transformation matrix can be obtained by aligning the intermediate picture to the specified specification picture to obtain transformation parameters of position coordinates, and constructing a transformation matrix according to the transformation parameters, and taking the transformation matrix as the second affine transformation matrix. The transformation parameters can be used to construct a translation transformation matrix T, a rotation transformation matrix R, and a scaling transformation matrix S. The translation transformation matrix T, the rotation transformation matrix R, and the scaling transformation matrix S are subjected to left multiplication operation to obtain the transformation matrix, and the transformation matrix is taken as the second affine transformation matrix.

[0118] It can be seen from the embodiment that the related art is to first scale the initial photo for processing, and the scaling processing causes loss of definition in the photo processing procedure, while the technical solution of the present application transforms the initial photo into an intermediate picture with unchanged resolution, which can reduce or avoid loss of definition in the photo processing procedure. In addition, the present application separately optimizes the definition of the portrait region and the background region of the intermediate picture, and then fuses the portrait region and the background region after the separate definition optimization to obtain a fused picture, which can perform different definition optimization according to the image characteristics of different regions in the picture, and further reduce or avoid loss of definition in the photo processing procedure. Therefore, the technical solution of the present application can improve the definition of the photo through optimization processing.

[0119] Figure 6 FIG. 4 is another structural schematic diagram of the photo processing apparatus shown in the embodiment of the present application.

[0120] Referring to Figure 6 A photo processing apparatus 50 includes a first transformation module 51, an optimization processing module 52, a fusion processing module 53, and a second transformation module 54.

[0121] The functions of the first transformation module 51, the optimization processing module 52, the fusion processing module 53, and the second transformation module 54 can be described with reference to the description in Figure 5 .

[0122] The first transformation module 51 can transform the initial photo into an intermediate picture with unchanged resolution through a first affine transformation matrix; and the second transformation module 54 can transform the fused picture into a specified specification picture through a second affine transformation matrix.

[0123] The optimization processing module 52 can include a first optimization sub-module 521 and a second optimization sub-module 522.

[0124] The first optimization sub-module 521 is configured to perform definition optimization processing on the portrait region of the intermediate picture according to a first optimization algorithm.

[0125] The second optimization sub-module 522 is configured to perform definition optimization processing on the background region of the intermediate picture according to a second optimization algorithm.

[0126] The first optimization algorithm can be a face blind restoration algorithm; and the second optimization algorithm can be a picture super-resolution algorithm. GFPGAN is the most advanced model of the face blind restoration algorithm, which uses a generated face prior to perform face blind restoration. The present application can use GFPGAN for face definition optimization but is not limited thereto. The present application can use a super-resolution algorithm model such as a RealESRGAN model for background definition optimization.

[0127] With regard to the apparatus in the above-described embodiments, the specific manner in which the respective modules perform operations has been described in detail in the embodiments related to the method, and will not be described in detail here.

[0128] Figure 7 FIG. 1 is a structural schematic diagram of an electronic device according to an embodiment of the present application.

[0129] Referring to FIG. 1, the electronic device 1000 includes a memory 1010 and a processor 1020. Figure 7

[0130] The processor 1020 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gates or transistor logic, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0131] ​The memory 1010 can include various types of storage units such as a system memory, a read-only memory (ROM), and a permanent storage device. Among them, the ROM can store static data or instructions required by the processor 1020 or other modules of the computer. The permanent storage device can be a rewritable storage device. The permanent storage device can be a non-volatile storage device that does not lose stored instructions and data even after the computer is powered off. In some embodiments, the permanent storage device employs a mass storage device (e.g., a magnetic or optical disk, a flash memory) as a permanent storage device. In some other embodiments, the permanent storage device can be a removable storage device (e.g., a floppy disk, an optical drive). The system memory can be a readable and writable storage device or a volatile readable and writable storage device such as a dynamic random access memory. The system memory can store some or all of the instructions and data required by the processor during runtime. In addition, the memory 1010 can include a combination of any computer readable storage media, including various types of semiconductor storage chips (e.g., DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), magnetic disks and / or optical disks. In some embodiments, the memory 1010 can include a readable and / or writable removable storage device such as a compact disc (CD), a read-only digital versatile disc (e.g., DVD-ROM, double-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (e.g., an SD card, a min SD card, a Micro-SD card, etc.), a magnetic floppy disk, etc. The computer readable storage medium does not include a carrier wave and an instantaneous electronic signal transmitted by wireless or wired transmission.

[0132] The memory 1010 stores executable code, which, when processed by the processor 1020, can cause the processor 1020 to perform part or all of the above-mentioned methods.

[0133] In addition, the method according to the present application can also be implemented as a computer program or a computer program product, which includes computer program code instructions for executing part or all of the steps of the above-mentioned methods of the present application.

[0134] Alternatively, the present application can also be implemented as a computer readable storage medium (or non-transitory machine readable storage medium or machine readable storage medium) having executable code (or computer program or computer instruction code) stored thereon, which, when executed by a processor of an electronic device (or a server, etc.), causes the processor to perform part or all of the steps of the above-mentioned methods according to the present application.

[0135] Having described various embodiments of the application, it is to be understood that the above description is meant not to limit and not to encompass all of the possible embodiments. Many modifications and variations of this application can be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. It is intended that the scope of the application be defined by the scope of the patent and by the claims as allowed by the patent office, which can include adaptations based on the description, equivalents, and / or substitutions of elements individually or collectively to the entire disclosure.

Claims

1. A method of photo processing, characterized by, The method comprises the following steps: transforming the initial photo into an intermediate picture with invariant resolution by a first affine transformation matrix; optimizing the resolution of the intermediate picture in portrait region and background region respectively; fusing the portrait region and the background region after the resolution optimization to obtain a fused picture; transforming the fused picture into a specified specification picture.

2. The method of claim 1, wherein, The first affine transformation matrix is obtained in the following way: aligning the initial photo to a reference picture to obtain transformation parameters of position coordinates; constructing a transformation matrix according to the transformation parameters, and setting the transformation matrix to obtain the first affine transformation matrix.

3. The method of claim 1, wherein: after the initial photo is transformed into an intermediate picture with invariant resolution by a first affine transformation matrix, if the picture resolution is greater than or equal to a set threshold, a first interpolation method is used for interpolation, and if the picture resolution is less than the set threshold, a second interpolation method is used for interpolation.

4. The method of claim 1, wherein, The resolution optimization of the intermediate picture in portrait region and background region respectively comprises: optimizing the resolution of the portrait region of the intermediate picture by a first optimization algorithm; optimizing the resolution of the background region of the intermediate picture by a second optimization algorithm.

5. The method of claim 1, wherein, The transformation of the fused picture into a specified specification picture comprises: transforming the fused picture into a specified specification picture by a second affine transformation matrix; wherein the second affine transformation matrix is obtained in the following way: aligning the intermediate picture to the specified specification picture to obtain transformation parameters of position coordinates; constructing a transformation matrix according to the transformation parameters, and taking the transformation matrix as the second affine transformation matrix.

6. A photograph processing apparatus characterized by comprising: The device comprises: a first transformation module for transforming an initial photo into an intermediate picture with invariant resolution by a first affine transformation matrix; an optimization processing module for optimizing the resolution of the intermediate picture transformed by the first transformation module in portrait region and background region respectively; a fusion processing module for fusing the portrait region and the background region after the resolution optimization by the optimization processing module to obtain a fused picture; a second transformation module for transforming the fused picture obtained by the fusion processing module into a specified specification picture.

7. The device of claim 6, wherein: the second transformation module transforms the fused picture into a specified specification picture by a second affine transformation matrix.

8. An electronic device, comprising: The device comprises: a processor; and a memory having executable code stored thereon, which, when executed by the processor, causes the processor to execute the method of any one of claims 1-5.

9. A computer-readable storage medium having executable code stored thereon, which, when executed by a processor of an electronic device, causes the processor to execute the method of any one of claims 1-5.

Images