Picture correction method, correction model training method and device, and electronic equipment

Abstract

Embodiments of the present application disclose a picture correction method, a correction model training method, a device and an electronic device. The method comprises: inputting a target picture obtained into a trained correction model, the correction model comprising a first-order correction model and a second-order correction model; obtaining feature information of the target picture through the first-order correction model, the feature information comprising a picture inversion category, a key point position and a screen category; if the screen category is that a screen exists in the target picture, performing preliminary correction on the target picture through the first-order correction model and based on the picture inversion category, and outputting a first corrected picture; performing secondary correction on the first corrected picture through the second-order correction model and based on the key point position of the target picture, and outputting a target corrected picture. The embodiments of the present application can improve the accuracy of correction of a leaked picture in a complex background environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, in particular to a picture correction method, a correction model training method, a device and an electronic device. BACKGROUND

[0002] With the continuous strengthening of national intellectual property awareness, people pay more and more attention to the protection of property rights. Especially in the field of images, if a picture does not have any property right proof, it may be misused and spread on the network, and the digital watermark technology is currently the main means to achieve the task of intellectual property protection. This technology embeds a watermark containing copyright information in the picture to achieve the purpose of protecting intellectual property rights.

[0003] Therefore, enterprises add their own logo watermark to the picture to protect their product rights. When the picture is leaked on the Internet, the intellectual property rights can be maintained by extracting the watermark of the leaked picture. In actual application, since the leaked pictures are mostly obtained by stealing, the leaked pictures mostly have screen frames, tilting and distortion, etc. In the process of extracting the watermark, the existing technology needs to manually correct the picture to facilitate the subsequent watermark extraction work.

[0004] However, manual correction can only handle simple correction of the leaked picture, and the accuracy of manual correction is low when the leaked picture is in a complex background environment. SUMMARY

[0005] The present application provides a picture correction method, a correction model training method, a device and an electronic device, which can improve the accuracy of correcting the leaked picture in a complex background environment.

[0006] In a first aspect, the present application provides a picture correction method, comprising:

[0007] inputting the obtained target picture into a trained correction model, wherein the correction model comprises a first-order correction model and a second-order correction model;

[0008] obtaining feature information of the target picture through the first-order correction model, wherein the feature information comprises picture inversion category, key point position and screen category;

[0009] if the screen category is that there is a screen in the target picture, performing preliminary correction on the target picture through the first-order correction model based on the picture inversion category, and outputting a first corrected picture;

[0010] performing secondary correction on the first corrected picture through the second-order correction model based on the key point position of the first corrected picture, and outputting a target corrected picture.

[0011] According to the technical scheme, the target picture is corrected by using the two-stage correction model, the first-stage correction model is used to position the key point positions of the target picture and extract the first key point positions, and the second-stage correction model is used to further correct the target picture according to the first key point positions, thereby improving the accuracy of the leaked picture correction in a complex background environment.

[0012] Optionally, the second correction of the first corrected picture based on the key point positions of the target picture by using the second correction model and output of the target corrected picture include:

[0013] cropping the first corrected picture according to the key point positions in the first corrected picture to obtain key point pictures corresponding to the key point positions in the first corrected picture;

[0014] inputting the key point pictures of the first corrected picture and the first corrected picture into the second correction model to output a second corrected picture;

[0015] performing affine transformation on the second corrected picture according to the key point positions in the second corrected picture to output the target corrected picture.

[0016] According to the technical scheme, the more accurate key point positions are extracted by combining the first corrected picture and the key point pictures corresponding to the key point positions in the first corrected picture, and the affine transformation is performed on the second corrected picture according to the more accurate key point positions, thereby improving the accuracy of the leaked picture correction.

[0017] In a second aspect of the present application, a correction model training method is provided, including:

[0018] constructing an initial correction model, the initial correction model including an initial first-stage correction model and an initial second-stage correction model;

[0019] obtaining a training picture set, the training picture set including a plurality of training pictures;

[0020] performing preprocessing and labeling processing on any training picture to obtain a first training picture;

[0021] inputting the first training picture into the initial first-stage correction model to output a second training picture;

[0022] inputting the second training picture into the initial second-stage correction model to output a target training picture;

[0023] calculating a loss value of the target training picture and determining whether the loss value is greater than a loss threshold value;

[0024] if the loss value is greater than the loss threshold, adjusting the initial correction model based on the loss value, and re-executing the steps of inputting the first training picture into the initial first-order correction model, outputting a second training picture, inputting the second training picture into the initial second-order correction model, and outputting a target training picture, until the loss value is less than or equal to the loss threshold, and each training picture in the training picture set is trained to obtain a trained correction model.

[0025] The above technical solution constructs an initial correction model, and iteratively trains the initial correction model through a training picture set, thereby improving the accuracy of the trained correction model in correcting a leaked picture.

[0026] Optionally, the initial first-order correction model includes a first convolutional network model and a feature pyramid network model, and the step of inputting the first training picture into the initial first-order correction model and outputting a second training picture includes:

[0027] inputting the first training picture into the first convolutional network model to output a screen category of the first training picture;

[0028] if the screen category of the first training picture is that a screen exists in the first training picture, inputting the first training picture into the feature pyramid network model to obtain a picture inversion category and a key point position of the first training picture; and correcting the first training picture according to the picture inversion category to output the second training picture.

[0029] The above technical solution combines the first convolutional network model and the feature pyramid network model, which can effectively slow down the problem of gradient vanishing in the process of the initial correction model, and the feature pyramid network model can better combine high-dimensional features and low-dimensional features, thereby improving the accuracy of extracting the screen category, the picture inversion category, and the key point position of the first training picture.

[0030] Optionally, the initial second-order correction model includes a second convolutional network model, and the step of inputting the second training picture into the initial second-order correction model and outputting a target training picture includes:

[0031] obtaining each key point position in the second training picture, and cropping the second training picture according to each key point position in the second training picture to obtain a key point picture corresponding to each key point position in the second training picture;

[0032] inputting the key point picture of the second training picture and the second training picture into the second convolutional network model to output a third training picture.

[0033] obtaining the positions of the key points in the third training picture, performing affine transformation on the third training picture according to the positions of the key points in the third training picture, and outputting the target training picture.

[0034] By adopting the above technical solution, the second training picture is cropped according to the positions of the key points in the second training picture, and the key point pictures corresponding to the positions of the key points are obtained, so that the second convolutional network model can extract the positions of the key points more finely according to the key point pictures.

[0035] Optionally, the cropping of the second training picture according to the positions of the key points in the second training picture to obtain the key point pictures corresponding to the positions of the key points in the second training picture comprises:

[0036] determining a cropping distance according to a preset first cropping proportion coefficient;

[0037] cropping the positions of the key points in the second training picture according to the coordinate sequence of the positions of the key points in the second training picture and the cropping distance to obtain the key point pictures corresponding to the positions of the key points in the second training picture.

[0038] By adopting the above technical solution, the cropping distance is determined by using the first cropping proportion coefficient, so that the cropping area of the key point pictures is determined, wherein the first cropping proportion coefficient can be iteratively converged in the process of training the initial correction model, so that the accuracy of the correction model in correcting the leaked pictures can be improved.

[0039] Optionally, the cropping of the second training picture according to the positions of the key points in the second training picture to obtain the key point pictures corresponding to the positions of the key points in the second training picture comprises:

[0040] determining a second cropping proportion coefficient according to the total area of the second training picture and the screen area;

[0041] cropping the positions of the key points in the second training picture respectively with the positions of the key points in the second training picture as the center according to the second cropping proportion coefficient to obtain the key point pictures corresponding to the positions of the key points in the second training picture.

[0042] By adopting the above technical solution, the second cropping proportion coefficient can be quickly determined according to the ratio of the total area of the second training picture to the area occupied by the screen, so that the cropping area of the key point pictures can be determined according to the second cropping proportion coefficient.

[0043] Optionally, the calculation of the loss value of the target training picture comprises:

[0044] According to the loss function formula, the loss value of the target training picture is calculated;

[0045] The loss function formula comprises:

[0046] Loss=Loss1+Loss2;

[0047] In the formula, Loss1 is the area loss value of the target training picture; Loss2 is the key point position loss value of the target training picture; and Loss is the loss value of the target training picture.

[0048] Wherein:

[0049]

[0050] In the formula, s i is the picture area of the i-th key point of the target training picture; s i ′ is the real area of the i-th key point picture of the target training picture; m is the number of key point pictures of the target training picture; s i =α(d1×d2);

[0051] In the formula, alpha is the first clipping ratio coefficient; d1 and d2 are respectively the Euclidean distance between the i-th key point position and the adjacent two key point positions in the target training picture;

[0052]

[0053] In the formula, n is the number of key point positions in the target training picture; (x i , y i ) is the coordinate of the i-th key point position of the target training picture, x i , y i are respectively the horizontal coordinate value and the vertical coordinate value in (x i , y i ); (x i ′ , y i ′ ) is the real coordinate of the i-th key point position of the target training picture, x i ′ , y i ′ are respectively the horizontal coordinate value and the vertical coordinate value of (x i ′ , y i ′ .

[0054] By using the above technical solution, the loss value is determined by the sum of the area loss value and the key point position loss value, so that the correction effect of the target training picture can be reflected according to the loss value.

[0055] In a third aspect of the present application, a picture correction device is provided, comprising:

[0056] a feature information determination module configured to input a target picture obtained by the obtaining module into a trained correction model, the correction model comprising a first-order correction model and a second-order correction model;

[0057] The feature information determination module is further configured to obtain feature information of the target picture by the first-order correction model, the feature information comprising a picture inversion category, a key point position, and a screen category;

[0058] a picture first-order correction module configured to perform preliminary correction on the target picture by the first-order correction model and based on the picture inversion category if the screen category indicates that a screen exists in the target picture, and output a first corrected picture;

[0059] a picture second-order correction module configured to perform secondary correction on the first corrected picture by the second-order correction model and based on the key point position of the target picture, and output a target corrected picture.

[0060] In a fourth aspect of the present application, an electronic device is provided, comprising a processor, a memory, a user interface, and a network interface, the memory being configured to store instructions, the user interface and the network interface being configured to communicate with other devices, and the processor being configured to execute the instructions stored in the memory.

[0061] In summary, the one or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages:

[0062] By using the technical solutions of the present application, the target picture is corrected by two-order correction models, the first-order correction model can locate the key point position of the target picture and extract the rough key point position, and the second-order correction model can further correct the target picture according to the rough key point position, thereby effectively improving the accuracy of correcting the leaked picture in a complex background environment. BRIEF DESCRIPTION OF DRAWINGS

[0063] Figure 1 is an application scenario diagram provided by the embodiments of the present application;

[0064] Figure 2 is a training system architecture diagram of an initial correction model provided by the embodiments of the present application;

[0065] Figure 3 is a flow diagram of an initial correction model training method provided by the embodiments of the present application;

[0066] Figure 4is a structural schematic diagram of an initial first-order correction model provided by an embodiment of the present application.

[0067] Figure 5 is a structural schematic diagram of an initial second-order correction model provided by an embodiment of the present application.

[0068] Figure 6 is a flow interaction diagram of a picture correction method provided by an embodiment of the present application.

[0069] Figure 7 is a structural schematic diagram of a picture correction device disclosed by an embodiment of the present application.

[0070] Figure 8 is a structural schematic diagram of an electronic device disclosed by an embodiment of the present application.

[0071] Legend: 701, feature information determination module; 702, picture first-order correction module; 703, picture second-order correction module; 800, electronic device; 801, processor; 802, memory; 803, user interface; 804, network interface; 805, communication bus. DETAILED DESCRIPTION

[0072] In order for those skilled in the art to better understand the technical solutions in the specification, the technical solutions in the specification will be clearly and completely described below in conjunction with the drawings in the embodiments of the specification. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. The same reference signs in the drawings represent the same or similar elements. Although various aspects of the embodiments are shown in the drawings, the drawings are not necessarily drawn to scale unless specifically indicated.

[0073] In the description of the embodiments of the present application, the words such as "for example" or "for instance" are used to represent an example, illustration or description. Any embodiment or design scheme described as "for example" or "for instance" in the embodiments of the present application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the words such as "for example" or "for instance" are intended to present the relevant concept in a specific manner.

[0074] In addition, in order to better illustrate the embodiments of the present application, numerous specific details are given in the specific embodiments below. Those skilled in the art should understand that the present application can also be implemented without some specific details. In some embodiments, methods, means, elements and circuits that are well known to those skilled in the art are not described in detail, in order to highlight the main ideas of the present application.

[0075] Please refer to Figure 1 , Figure 1 application embodiments provided by an embodiment of the present application. As shown in FIG. 1, the application embodiments are applied to a picture correction device 800.Figure 1 As shown, to protect their product rights, companies typically add watermarks with their own logo to product images. When these images are leaked online, companies can extract the watermarks to protect their rights. However, in practice, since most leaked images are obtained through surreptitious means, such as... Figure 1 As shown, the leaked image may have screen borders, tilt, and distortion, which would cause corresponding deformation of the watermark, making it difficult to identify. Therefore, the leaked image needs to be corrected before extracting the watermark.

[0076] This application provides an image correction method, apparatus, and electronic device. By inputting a leaked image into the image correction apparatus, screen borders present in the leaked image can be removed, and distortions such as tilting and warping in the leaked image can be corrected. The output is as follows: Figure 1 The corrected leaked image shown makes it easy to clearly identify the watermark content within the leaked image.

[0077] The image correction device includes, but is not limited to, servers, Android devices, Apple's iOS devices, personal computers (PCs), World Wide Web (WEB) devices, virtual reality (VR) devices, and augmented reality (AR) devices. Furthermore, this application embodiment does not limit the device from which the leaked image is input. The image correction device stores an application program for an image correction method, which in this application embodiment is primarily implemented by a correction model.

[0078] Having introduced the application scenarios above, to help those skilled in the art better understand the principle of this image correction method, the training process of the correction model will be explained below. Please refer to... Figure 2 , Figure 2 This diagram illustrates an initial calibration model training system architecture provided in an embodiment of this application. Figure 2 As shown, the initial correction model includes a preprocessing module, an initial first-order correction model, and an initial second-order correction model. The initial first-order correction model includes a first convolutional network model, a feature pyramid network model, and a first-order correction module. The initial second-order correction model includes an image cropping module, a second convolutional network model, a second-order correction module, and a post-processing module.

[0079] Here, "training images" refers to images leaked on the internet. In this application's embodiments, it mainly refers to images leaked through means such as surreptitious photography, as well as screenshots from leaked videos, etc. Figure 1 As shown, training images may contain screen borders, inversions, distortions, etc.Figure 2 As shown, the training picture can be input into the initial correction model for correction, and the corrected training picture is output. By judging whether the corrected training picture meets the standard, if the corrected training picture does not meet the standard, the parameters in the initial correction model can be adjusted in time, and the training picture is re-substituted into the initial correction model for repeated training until the corrected training picture meets the standard, thereby completing the training of the initial correction model and obtaining the correction model.

[0080] Specifically, after the training picture is input into the preprocessing module, the preprocessing module can adjust the training picture to a picture suitable for the network size in the initial first-order correction model, and preliminarily mark the key point positions in the training picture to obtain a first training picture, and output the first training picture to the first convolutional network model. The first convolutional network model can extract the key point positions in the first training picture and output them to the first correction module for first correction. If the first training picture is reversed, corresponding reversal adjustment can also be performed to obtain a second training picture, which is output to the initial second-order correction model.

[0081] Further, the picture cropping module in the initial second-order correction model receives the second training picture, and crops the corresponding key point pictures according to the key point positions in the second training picture, and outputs the cropped key point pictures and the second training picture to the second convolutional network model. The second convolutional network model extracts the key point positions of the second training picture more finely according to the key point pictures to obtain a third training picture, and outputs the extracted key point positions to the second correction module. The second correction module can perform affine transformation on the third training picture according to the key point positions in the third training picture, and output the second-corrected picture to the post-processing module. The post-processing module restores the size and color of the picture, and the corrected training picture can be output.

[0082] The above describes the training system architecture of the initial correction model and the interaction of each module under the system architecture. On the basis of the above embodiments, further Figure 3 , an initial correction model training method flowchart is proposed. The method can be implemented by relying on a computer program, relying on a single-chip microcomputer, or running on a picture correction device. Specifically, the method includes steps 301 to 306, which are as follows:

[0083] Step 301: Construct an initial correction model and obtain a training picture set.

[0084] The training picture set refers to a training set used for model training. The training picture set can include multiple training pictures. The initial correction model can be trained by using the training picture set. The initial correction model mainly extracts the key point positions of the training pictures, so as to correct the training pictures according to the key point positions. In the embodiment of the present application, the initial correction model can include an initial first-order correction model and an initial second-order correction model. The initial first-order correction model can roughly extract the key point positions, so as to input the extracted key point positions into the initial second-order correction model for fine secondary extraction, to obtain more accurate key point positions.

[0085] For any one training picture in the training picture set, steps 302 to 306 are performed.

[0086] Step 302: The training picture is preprocessed and labeled to obtain a first training picture.

[0087] For example, since the training picture is obtained by different scenes and different shooting angles, or is a picture cut from a video, the training picture needs to be denoised first. Then, according to the picture format requirement of the initial correction model, the size of the training picture is converted to a format size suitable for the preset format size of the initial correction model, and the training picture with color is converted to a gray picture.

[0088] Further, the key point positions in the training picture need to be labeled. The label tool labelme can be used for manual labeling to obtain accurate key point positions, so as to verify whether the loss rate of the corrected training picture meets the standard in the later period.

[0089] The feature information mainly includes picture reversal category, key point position, and screen category. The picture reversal category is whether the training picture is upside down or not upside down. The screen category is whether the training picture has a screen or not. The screen can be understood as the frame of the display shown in FIG. 1. The key point position can be understood as the vertex of the screen of the training picture. Since the training picture is obtained by shooting, the training picture is not complete, and the number of vertices is not unique, which is 1 to 4. Figure 1

[0090] It should be noted that the above-mentioned pre-processing and labeling method of the training picture is only exemplary, and the execution order of the two processes is not limited. In the embodiment of the present application, the training picture after pre-processing and labeling is defined as a first training picture.

[0091] Step 303: The first training picture is input into the initial first-order correction model, and a second training picture is output.

[0092] For example, as shown in FIG. 2, the first training picture is input into the initial first-order correction model, and the second training picture is output.​Figure 2 As shown, the first-order correction model mainly includes a first convolutional network model and a first-order correction module in the embodiments of the present application. The first training picture is input into the first convolutional network model, the first convolutional network model can extract the feature information of the first training picture, and output the feature information to the first-order correction module. The first-order correction module performs first-order correction on the first training picture according to the feature information, obtains a second training picture with no up-down reversal, no screen in the first training picture, and containing key point positions, and outputs the second training picture to the second-order correction module.

[0093] In a possible implementation, referring to Figure 4 which shows a structural schematic diagram of an initial first-order correction model, step 303 can further include the following steps:

[0094] Step 401: input the first training picture into the first convolutional network model, and output the screen category of the first training picture.

[0095] Exemplarily, a convolutional network model (CNN) is a multi-layer neural network, each layer of which is composed of multiple two-dimensional planes. The CCN can effectively extract the feature information in the picture, and is mainly used to solve the classification problem of image recognition. In the embodiments of the present application, the screen category in the feature information of the first training picture can be extracted by CNN.

[0096] Step 402: if the screen category of the first training picture is that there is a screen in the first training picture, input the first training picture into the feature pyramid network model, and obtain the picture reversal category and the key point position of the first training picture.

[0097] In a possible implementation, after determining that the screen category of the first training picture is that there is a screen in the first training picture, the first training picture is input into the feature pyramid network model to further obtain the reversal category and the key point position of the first training picture.

[0098] In another possible implementation, if it is determined that the screen category of the first training picture is that there is no screen in the first training picture, the training process can be directly exited.

[0099] The feature pyramid network model (FPN) is a network structure that uses the multi-scale feature maps inherent in the deep convolutional neural network, adds lateral connections and up-sampling, and constructs a feature pyramid with high-level semantic information of different scales with minimal additional computational cost. The FPN can fuse the features on different feature maps, and then make a prediction on the fused feature maps.

[0100] As shown in Figure 4 After the training picture is preprocessed and labeled, it is input into the CNN to build a bottom-up pyramid network structure (Res2, Res3, Res4, Res5). C2 is obtained by copying Res2, performing 2 times upsampling operation on c2, performing dimension reduction on Res3 through 1*1 convolution, and then adding corresponding elements of the two to realize the collection of high and low layer features and obtain c3. C4 and c5 are obtained in the same way. Finally, 3*3 convolution operation is performed on c2, c3, c4 and c5 to obtain the final prediction result.

[0101] Further, the prediction result can be represented as a one-dimensional array carrying 3 identification bits (identification bit 1, identification bit 2, identification bit 3) of feature information. By judging the 3 identification bits, the picture reversal category and the screen category of the first training picture can be determined, and the key point position of the first training picture can be determined by combining the regression algorithm.

[0102] It should be noted that the above is only an exemplary description of the network structure of the FPN. The FPN network can also be other neural networks, and the corresponding network structure can also be other network structures. The network structure is not limited in the present application.

[0103] Step 403: correcting the first training picture according to the picture reversal category, and outputting a second training picture.

[0104] For example, if the picture reversal category of the first training picture is upside down, that is, the deflection angle of the first training picture is greater than the preset deflection angle. At this time, the deflection angle of the first training picture can be corrected by rotating it, so as to obtain the second training picture, wherein the second training picture carries the key point position.

[0105] Step 304: inputting the second training picture into the initial second-order correction model to output a target training picture.

[0106] As shown in Figure 2 The initial second-order correction model mainly includes a picture cropping module, a second convolutional neural network model, a secondary correction module and a post-processing module. In the embodiment of the present application, the initial second-order correction model mainly crops the key point positions in the second training picture through the cropping module, so that the second convolutional neural network model can correct the second training picture according to the key point pictures obtained by cropping to further determine more accurate key point positions. The secondary correction module can correct the second training picture according to the key point positions to obtain the target training picture.

[0107] In one possible implementation, the present embodiment provides two key point picture cropping methods. Please refer to Figure 5As shown in FIG. 4, which shows a structural diagram of an initial second-order correction model, the clipping module can include a first key point position clipping module and a second key point position clipping module. Further, step 304 can further include the following steps:

[0108] Step 501: Obtain the positions of the key points in the second training picture, and clip the second training picture according to the positions of the key points in the second training picture to obtain the key point pictures corresponding to the positions of the key points in the second training picture.

[0109] In a feasible example, the specific steps of implementing the process by the first key point position clipping module are as follows: determine the clipping distance according to the preset first clipping proportion coefficient. Clip the positions of the key points in the second training picture based on the coordinate order of the key points and the clipping distance to obtain the key point pictures corresponding to the positions of the key points in the second training picture.

[0110] The clipping distance refers to the distance obtained by multiplying the Euclidean distance between two key points by the first clipping proportion coefficient. The coordinate order can be understood in the embodiments of the present application as the order of the positions of the key points. The order is the clockwise order of the positions of the key points. If the training picture is corrected in the initial first-order correction model, the order needs to be refreshed. After determining the clipping distance of the positions of the key points and the positions of the adjacent key points according to the preset first clipping proportion coefficient, the positions of the key points in the second training picture can be clipped according to the coordinate order of the key points to obtain the corresponding key point pictures.

[0111] For example, the second training picture has four key points (a, b, c, and d in clockwise order). When a is clipped, the Euclidean distances d1 and d2 between a and b and between a and d are calculated, respectively. The corresponding clipping distances are a*d1 and a*d2. Thus, a is clipped as the center with a clipping area of (a*d1)*(a*d2) to obtain the key point picture corresponding to a. The clipping principles of the positions of the other key points are the same as above and will not be described in detail.

[0112] Further, the range of the first clipping proportion coefficient a is set as a∈(0, 0.5] during the training of the initial correction model. The first clipping proportion coefficient a presents a decreasing function change trend and gradually decreases until convergence during the training of the initial model.

[0113] In another possible example, the specific steps of the process are implemented by a second key point position cropping module, which includes: determining a second cropping ratio coefficient according to the total area of the second training picture and the area ratio of the screen area, and cropping each key point position in the second training picture according to the second cropping ratio coefficient and taking each key point position in the second training picture as the center to obtain a key point picture corresponding to each key point position in the second training picture.

[0114] Step 502: input the key point picture of the second training picture and the second training picture into the second convolutional network model to output a third training picture.

[0115] For example, after the key point positions of the second training picture are cropped by the cropping module, a key point picture corresponding to each key point position can be obtained, so that the key point picture of the second training picture and the second training picture can be input into the second convolutional network model, so that more accurate key point positions can be extracted, and a third training picture can be obtained. For details of the process, please refer to steps 401 to 402, which will not be repeated here.

[0116] Step 503: obtain each key point position in the third training picture, perform affine transformation on the third training picture according to each key point position in the third training picture, and output a target training picture.

[0117] The affine transformation refers to a linear transformation and a translation of a vector space in geometry, so as to transform into another vector space. When the affine transformation is applied to picture correction, it can be understood that the picture is translated, rotated and transformed through a series of geometric transformations.

[0118] For example, the second convolutional network model inputs the third training picture into the secondary correction module, the secondary correction module can perform affine transformation on the third training picture according to the key point positions carried in the third training picture to obtain a corrected third training picture, and output the corrected third training picture to the post-processing module. After the post-processing module restores the corrected third training picture to the initial size and color of the training picture, the target training picture can be obtained.

[0119] Step 305: calculate the loss value of the target training picture, and determine whether the loss value is greater than a loss threshold.

[0120] The loss value Loss is a value reflecting the correction ability of the initial training model, which can be specifically expressed as the sum of an area loss value Loss1 and a key point position loss value Loss2. The loss value of the target training picture can be calculated according to the loss function formula.

[0121] The loss function formula includes:

[0122] Loss = Loss1 + Loss2.

[0123] In the formula, Loss1 is the target training picture area loss value; Loss2 is the target training picture key point position loss value; and Loss is the loss value of the target training picture.

[0124] Wherein:

[0125]

[0126] In the formula, s i is the i-th key point picture area of the target training picture; s i ′ is the real area of the i-th key point picture of the target training picture; m is the number of key point pictures of the target training picture; wherein, s i ′ is obtained by marking the key point picture corresponding to the key point position of the target training picture;

[0127] s i = a (d1 x d2);

[0128] In the formula, a is the first clipping ratio coefficient; d1 and d2 are respectively the Euclidean distances between the i-th key point position and the adjacent two key point positions in the target training picture;

[0129]

[0130] In the formula, Loss2 is the key point position loss value; n is the number of key point positions in the target training picture; (x i , y i ) is the coordinate of the i-th key point position of the target training picture, x i , y i are respectively the horizontal coordinate value and the vertical coordinate value in (x i , y i ); (x i ′ , y i ′ ) is the real coordinate of the i-th key point position of the target training picture, x i ′ , y i ′ are respectively the horizontal coordinate value and the vertical coordinate value of (x i ′ , y i ′ ;

[0131] Wherein, (x i ′ , y i ′) is obtained.

[0132] Step 306: If the loss value is greater than the loss threshold, the initial correction model is adjusted based on the loss value, and the steps of inputting the first training picture into the initial first-order correction model, outputting the second training picture, inputting the second training picture into the initial second-order correction model, outputting the target training picture, calculating the loss value of the target training picture, and judging whether the loss value is greater than the loss threshold are re-executed until the loss value is less than or equal to the loss threshold, and each training picture in the training picture set is trained to obtain a trained correction model.

[0133] For example, after the loss value of the target training picture is calculated, the loss value can be compared with the loss threshold to determine whether the initial correction model is accurate in correcting the training picture. If the loss value is greater than the loss threshold, it means that the process is not accurate, and the parameters of the initial correction model are adjusted based on the loss value, and the first training picture of the training picture is re-input into the first-order correction model for iterative correction until the loss value is less than or equal to the loss threshold. When all training pictures in the training picture set are trained, it is determined that the initial correction model is trained.

[0134] The above embodiment describes the training process of the initial correction model. Based on the above embodiment, the process of correcting the target picture by the trained correction model will be described as follows. Please refer to Figure 6 which shows a flow interaction diagram of a picture correction method provided by an embodiment of the present application. The following will be described in two stages. The stages do not need to be continuous, but only need to be executed at a preset time period.

[0135] Stage one: first-order correction process.

[0136] Step 601: The terminal inputs a target picture into a first-order correction model of a correction model.

[0137] Step 602: The first-order correction model pre-processes the target picture, inputs the pre-processed target picture into a first convolutional network model, and outputs the screen category thereof. The first-order correction model includes the first convolutional network model and a feature pyramid network model.

[0138] Step 603: It is judged whether the screen category of the target picture is that there is a screen in the target picture.

[0139] Step 604: If the screen category of the target picture is not that there is a screen in the target picture, the target picture is taken as a first correction picture, output to a second-order correction model, and step 608 is executed.

[0140] Step 605: input the target picture into the feature pyramid network model to obtain the picture inversion category and the key point position of the target picture.

[0141] Step 606: correct the target picture according to the picture inversion category to obtain a first corrected picture.

[0142] Step 607: input the first corrected picture into the second-order correction model.

[0143] Phase two: the second-order correction process.

[0144] Step 608: according to the key point positions in the first corrected picture, the first corrected picture is cropped to obtain the key point pictures corresponding to the key point positions in the first corrected picture.

[0145] Step 609: input the key point pictures of the first corrected picture and the first corrected picture into the second convolution network model to obtain a second corrected picture.

[0146] Step 610: according to the key point positions in the second corrected picture, the second corrected picture is subjected to affine transformation and post-processing to obtain a target corrected picture.

[0147] Step 611: output the target corrected picture to the terminal.

[0148] The above process is described in detail in the training process of the initial correction model, i.e., steps 301-306, which will not be described in detail here.

[0149] The following is an embodiment of the system of the present application, which can be used to execute the method embodiments of the present application. For details not disclosed in the system embodiments of the present application, please refer to the method embodiments of the present application.

[0150] Referring to Figure 7 The picture correction device provided by the embodiment of the present application can include a feature information determination module 701, a picture first-order correction module 702, and a picture second-order correction module 703, wherein:

[0151] The feature information determination module 701 is configured to input the obtained target picture into the trained correction model, wherein the correction model includes a first-order correction model and a second-order correction model.

[0152] The feature information determination module 701 is further configured to obtain the feature information of the target picture through the first-order correction model, wherein the feature information includes a picture inversion category, a key point position, and a screen category.

[0153] The picture first-order correction module 702 is configured to perform preliminary correction on the target picture by using the first-order correction model and based on the picture inversion category if the screen category is that a screen exists in the target picture, and output a first corrected picture.

[0154] The picture second-order correction module 703 is configured to perform secondary correction on the first corrected picture by using the second-order correction model and based on the key point positions of the first corrected picture, and output a target corrected picture.

[0155] In the above embodiment, as an optional embodiment, the picture second-order correction module 703 can further include a key point picture determining unit, a third corrected picture determining unit and a target corrected picture determining unit, wherein:

[0156] The key point picture determining unit is configured to crop the first corrected picture according to the key point positions in the first corrected picture, to obtain a key point picture corresponding to each key point position in the first corrected picture.

[0157] The third corrected picture determining unit is configured to input the key point picture of the first corrected picture and the first corrected picture into the second-order correction model, and output a second corrected picture.

[0158] The target corrected picture determining unit is configured to perform affine transformation on the second corrected picture according to the key point positions in the second corrected picture, and output the target corrected picture.

[0159] In the above embodiment, as an optional embodiment, the picture correction device can further include an initial correction model constructing module, a training picture set obtaining module, a first training picture determining module, a first-order correction model training module, a second-order correction model training module, a training picture loss value calculating module and a correction model repeated training module, wherein: the initial correction model constructing module is configured to construct an initial correction model, and the initial correction model includes an initial first-order correction model and an initial second-order correction model.

[0160] The training picture set obtaining module is configured to obtain a training picture set, and the training picture set includes a plurality of training pictures.

[0161] The first training picture determining module is configured to perform preprocessing and labeling processing on any training picture to obtain a first training picture.

[0162] The first-order correction model training module is configured to input the first training picture into the initial first-order correction model, and output a second training picture.

[0163] The second-order correction model training module is configured to input the second training picture into the initial second-order correction model to output a target training picture.

[0164] The training picture loss value calculation module is configured to calculate a loss value of the target training picture and determine whether the loss value is greater than a loss threshold value.

[0165] The correction model repeated training module is configured to, if the loss value is greater than the loss threshold value, adjust the initial correction model based on the loss value, and re-perform the steps of inputting the first training picture into the initial first-order correction model to output a second training picture, inputting the second training picture into the initial second-order correction model to output a target training picture, calculating a loss value of the target training picture, and determining whether the loss value is greater than a loss threshold value, until the loss value is less than or equal to the loss threshold value and each training picture in the training picture set is trained to obtain a trained correction model.

[0166] In the above embodiment, as an optional embodiment, the first-order correction model training module can further include a first training picture feature information output unit, a first training picture primary correction unit, and a first training picture secondary correction unit, wherein:

[0167] The first training picture feature information output unit is configured to input the first training picture into the first convolutional network model to output a screen category of the first training picture.

[0168] The first training picture primary correction unit is configured to, if the screen category of the first training picture is that there is a screen in the first training picture, input the first training picture into the feature pyramid network model to obtain a picture inversion category and a key point position of the first training picture.

[0169] The first training picture secondary correction unit is configured to correct the first training picture according to the picture inversion category to output the second training picture.

[0170] In the above embodiment, as an optional embodiment, the second-order correction model training module can further include a second training picture cropping unit, a second training picture correction unit, and a third training picture affine transformation unit, wherein:

[0171] The second training picture cropping unit is configured to obtain each key point position in the second training picture, crop the second training picture according to each key point position in the second training picture to obtain a key point picture corresponding to each key point position in the second training picture.

[0172] a second training picture correction unit configured to input the key point picture of the second training picture and the second training picture into the second convolutional neural network model, and output a third training picture;

[0173] a third training picture affine transformation unit configured to obtain the positions of the key points in the third training picture, perform affine transformation on the third training picture according to the positions of the key points in the third training picture, and output the target training picture.

[0174] In the above embodiment, as an optional embodiment, the second training picture cropping unit can further include a cropping distance determination subunit and a first key point picture determination subunit, wherein:

[0175] the cropping distance determination subunit is configured to determine the cropping distance according to a preset first cropping proportion coefficient;

[0176] the first key point picture determination subunit is configured to crop the positions of the key points in the second training picture based on the coordinate sequence of the positions of the key points in the second training picture and the cropping distance, and obtain the key point pictures corresponding to the positions of the key points in the second training picture.

[0177] In the above embodiment, as another optional embodiment, the second training picture cropping unit can further include a second cropping proportion coefficient determination subunit and a second key point picture determination subunit, wherein:

[0178] the second cropping proportion coefficient determination subunit is configured to determine a second cropping proportion coefficient according to the total area of the second training picture and the screen area;

[0179] the second key point picture determination subunit is configured to crop the positions of the key points in the second training picture respectively with the positions of the key points in the second training picture as the center according to the second cropping proportion coefficient, and obtain the key point pictures corresponding to the positions of the key points in the second training picture.

[0180] In the above embodiment, as an optional embodiment, the training picture loss value calculation module can further include a loss value calculation unit, wherein:

[0181] the loss value calculation unit is configured to calculate the loss value of the target training picture according to a loss function formula;

[0182] the loss function formula includes:

[0183] Loss = Loss1 + Loss2;

[0184] In the formula, Loss1 is the area loss value of the target training picture; Loss2 is the key point position loss value of the target training picture; and Loss is the loss value of the target training picture.

[0185] In the formula, s

[0186]

[0187] In the formula, s i is the i-th key point picture area of the target training picture; s i ′ is the real area of the i-th key point picture of the target training picture; m is the number of key point pictures of the target training picture; s i =α(d1×d2);

[0188] In the formula, α is a first clipping ratio coefficient; d1 and d2 are respectively the Euclidean distances between the i-th key point position and the adjacent two key point positions in the target training picture;

[0189]

[0190] In the formula, Loss2 is the key point position loss value; n is the number of key point positions in the target training picture; (x i , y i ) is the coordinate of the i-th key point position of the target training picture, x i , y i are respectively the horizontal coordinate value and the vertical coordinate value in (x i , y i ); (x i ′ , y i ′ ) is the real coordinate of the i-th key point position of the target training picture, x i ′ , y i ′ are respectively the horizontal coordinate value and the vertical coordinate value of (x i ′ , y i ′ ).

[0191] It should be noted that the apparatus provided in the above embodiments is only used as an example to illustrate the division of the above functional modules in realizing its functions, and in actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the above described functions. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0192] The application also discloses an electronic device. Referring to Figure 8 Figure 8 is a structural schematic diagram of an electronic device disclosed by the embodiment of the application. The electronic device 800 can include at least one processor 801, at least one network interface 804, a user interface 803, a memory 802 and at least one communication bus 805.

[0193] The communication bus 805 is configured to realize the connection and communication between the components.

[0194] The user interface 803 can include a display screen (Display) and a camera (Camera), and the optional user interface 803 can further include a standard wired interface and a wireless interface.

[0195] The network interface 804 can optionally include a standard wired interface and a wireless interface (such as a WI-FI interface).

[0196] The processor 801 can include one or more processing cores. The processor 801 connects various parts in the server through various interfaces and lines, executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 802 and calling data stored in the memory 802. Optionally, the processor 801 can be realized in at least one of the hardware forms of a digital signal processing (DSP), a field-programmable gate array (FPGA) and a programmable logic array (PLA). The processor 801 can be integrated with a combination of one or more of a central processing unit (CPU), a graphics processing unit (GPU) and a modem. The CPU is mainly used to process an operating system, a user interface graph and an application program; the GPU is used to render and draw the content to be displayed on the display screen; and the modem is used to process wireless communication. It can be understood that the above-mentioned modem can also not be integrated into the processor 801, but can be realized by a separate chip.

[0197] ​The memory 802 may include random access memory (RAM) or read-only memory. Optionally, the memory 802 may include a non-transitory computer-readable storage medium. The memory 802 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 802 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory 802 may also be at least one storage device located remotely from the aforementioned processor 801. (Refer to...) Figure 8 The memory 802, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and an application program for an image correction method and a correction model training method.

[0198] exist Figure 8 In the illustrated electronic device 800, the user interface 803 is mainly used to provide an input interface for the user and acquire user input data; while the processor 801 can be used to call an application program stored in the memory 802 that contains an image correction method and a correction model training method. When executed by one or more processors 801, the electronic device 800 performs one or more of the methods described in the above embodiments. It should be noted that, for the foregoing method embodiments, for the sake of simplicity, they are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0199] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0200] In several embodiments provided in the present application, it should be understood that the disclosed apparatus can be implemented in other manners. For example, the division of the apparatus embodiments is merely illustrative, and the division of units can be changed according to actual conditions, such as a combination or integration of some units, or a deletion of some features, or an addition of some features. In addition, the coupling or direct coupling or communication connection between the shown or discussed units can be indirect coupling or communication connection through some interfaces, devices or units, and can be in electrical, mechanical or other forms.

[0201] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one place or distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0202] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0203] If the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable memory. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a memory and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the embodiments of the present application. The aforementioned memory includes: a U disk, a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0204] The above is only exemplary embodiments of the present disclosure, and cannot limit the scope of the present disclosure. That is, any equivalent changes and modifications made in accordance with the teachings of the present disclosure are still within the scope of the present disclosure. Those skilled in the art will easily think of other embodiments of the present disclosure after considering the specification and the true disclosure.

[0205] The present application is intended to cover any variations, uses or adaptive changes of the present disclosure that follow the general principles of the present disclosure and include common knowledge or conventional technical means in the technical field not described in the present disclosure. The specification and examples are only considered as exemplary, and the scope and spirit of the present disclosure are defined by the claims.

Claims

1. An image correction method, characterized in that, include: The acquired target image is input into the trained correction model, which includes a first-order correction model and a second-order correction model. The first-order correction model is used to obtain the feature information of the target image, which includes the image inversion category, key point positions, and screen category. If the screen category is that there is a screen in the target image, then the target image is initially corrected by the first-order correction model and based on the image inversion category, and a first corrected image is output. The second-order correction model is used to perform secondary correction on the first corrected image based on the key point positions of the first corrected image, and output a target corrected image. This includes: cropping the first corrected image according to the key point positions in the first corrected image to obtain key point images corresponding to each key point position in the first corrected image, including: determining the cropping distance according to a preset first cropping ratio coefficient, wherein the cropping distance is the distance obtained by multiplying the Euclidean distance between two key points by the first cropping ratio coefficient; and cropping each key point position based on the coordinate order of each key point position and the cropping distance to obtain key point images. Input the key point image of the first corrected image and the first corrected image into the second-order correction model, and output the second corrected image; Based on the positions of each key point in the second corrected image, an affine transformation is performed on the second corrected image to output the target corrected image.

2. A method for training a calibration model, characterized in that, include: Construct an initial calibration model, which includes an initial first-order calibration model and an initial second-order calibration model, wherein the initial second-order calibration model includes a second convolutional network model; Obtain a training image set, which includes multiple training images; Preprocessing and annotation of any of the training images to obtain the first training image; The first training image is input into the initial first-order correction model, and the second training image is output. The process of inputting the second training image into the initial second-order correction model and outputting the target training image includes: obtaining the positions of each key point in the second training image, cropping the second training image according to the positions of each key point in the second training image, and obtaining the key point image corresponding to each key point position in the second training image. The key point images of the second training image and the second training image are input into the second convolutional network model, and the third training image is output. Obtain the positions of each key point in the third training image, perform an affine transformation on the third training image based on the positions of each key point, and output the target training image. Calculate the loss value of the target training image and determine whether the loss value is greater than the loss threshold; If the loss value is greater than the loss threshold, the initial calibration model is adjusted based on the loss value, and the process of inputting the first training image into the initial first-order calibration model and outputting the second training image is repeated, and the second training image is input into the initial second-order calibration model and outputting the target training image is repeated; the loss value of the target training image is calculated, and it is determined whether the loss value is greater than the loss threshold, until the loss value is less than or equal to the loss threshold, and all the training images in the training image set have been trained, thus obtaining the trained calibration model.

3. The calibration model training method according to claim 2, characterized in that, The initial first-order correction model includes a first convolutional network model and a feature pyramid network model. The step of inputting the first training image into the initial first-order correction model and outputting a second training image includes: Input the first training image into the first convolutional network model and output the screen category of the first training image; If the screen category of the first training image is that there is a screen in the first training image, then the first training image is input into the feature pyramid network model to obtain the image inversion category and key point position of the first training image. Based on the inverted category of the image, the first training image is corrected, and the second training image is output.

4. The calibration model training method according to claim 2, characterized in that, The step of cropping the second training image based on the positions of each key point in the second training image to obtain key point images corresponding to the positions of each key point in the second training image includes: The cutting distance is determined based on the preset first cutting ratio coefficient; Based on the coordinate order of each key point position in the second training image and the cropping distance, the key point positions in the second training image are cropped to obtain key point images corresponding to each key point position in the second training image.

5. The calibration model training method according to claim 2, characterized in that, The step of cropping the second training image based on the positions of each key point in the second training image to obtain key point images corresponding to the positions of each key point in the second training image includes: The second cropping ratio coefficient is determined based on the total area of ​​the second training image and the screen area. Based on the second cropping ratio coefficient, each key point in the second training image is cropped with the key point position as the center to obtain the key point image corresponding to each key point position in the second training image.

6. The calibration model training method according to claim 2, characterized in that, The calculation of the loss value of the target training image includes: Calculate the loss value of the target training image according to the loss function formula; The loss function formula includes: Loss = Loss1 + Loss2; In the formula, Loss1 is the area loss value of the target training image; Loss2 is the key point position loss value of the target training image; and Loss is the loss value of the target training image. in: ； In the formula, The area of ​​the i-th keypoint in the target image is used for training. The true area of ​​the i-th keypoint image in the target training image; m is the number of keypoint images in the target training image. ; In the formula, This is the first cutting ratio coefficient; , are the Euclidean distances between the i-th keypoint position and the positions of the two adjacent keypoints in the target training image, respectively. ； In the formula, n is the number of keypoint locations in the target training image; The coordinates of the i-th keypoint in the target training image are given. They are respectively ( The x and y coordinates in the diagram; ( ) represents the true coordinates of the i-th keypoint in the target training image. They are respectively ( The x and y coordinates of ).

7. An image correction device, characterized in that, include: The feature information determination module (701) is used to input the acquired target image into the trained correction model, which includes a first-order correction model and a second-order correction model; The feature information determination module (701) is also used to obtain the feature information of the target image through the first-order correction model. The feature information includes the image inversion category, key point position and screen category. The first-order image correction module (702) is used to perform preliminary correction on the target image by means of the first-order correction model and based on the image inversion category if the screen category is that there is a screen in the target image, and output the first corrected image. The image second-order correction module (703) is used to perform secondary correction on the first correction image based on the key point positions of the target image using the second-order correction model, and output the target correction image. This includes: cropping the first correction image according to the key point positions in the first correction image to obtain key point images corresponding to each key point position in the first correction image; determining the cropping distance according to a preset first cropping ratio coefficient, where the cropping distance is the Euclidean distance between two key points multiplied by the first cropping ratio coefficient; and cropping each key point position based on the coordinate order of each key point position and the cropping distance to obtain key point images. Input the key point image of the first corrected image and the first corrected image into the second-order correction model, and output the second corrected image; Based on the positions of each key point in the second corrected image, an affine transformation is performed on the second corrected image to output the target corrected image.

8. An electronic device, characterized in that, include: The electronic device (800) comprises a processor (801), a memory (802), a user interface (803), and a network interface (804), wherein the memory (802) is used to store instructions, the user interface (803) and the network interface (804) are used to communicate with other devices, and the processor (801) is used to execute the instructions stored in the memory (802) to cause the electronic device (800) to perform the method as described in any one of claims 1-6.

Images