Image processing method and device and storage medium

Abstract

The invention relates to an image processing method and device and a storage medium, and the method comprises the steps: obtaining an original image, and determining an element region of a target element in the original image; obtaining geometric parameters of the original image and vertex coordinates of the element region; determining a reduction geometric proportion of the element region according to the vertex coordinates and the geometric parameters; and based on the reduction geometric proportion, carrying out proportion reduction on the element region to obtain a target element region. Therefore, the reduction geometric proportion of the element area is determined based on the geometric parameters of the original image and the vertex coordinates of the element area, so that the element area is subjected to proportion reduction, and the proportion distortion of the reduced element area can be avoided.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an image processing method, device and storage medium. Background Art

[0002] In recent years, portable devices (such as mobile phones) are often used to capture various elements (such as paper documents, cards, screens, blackboards, etc.) to obtain images for recording or sharing various elements. In order to better record or share various elements, it is often necessary to extract element areas (such as document areas) in the image, and crop and transform the element areas.

[0003] Due to perspective distortion, the cropped and transformed element area cannot reflect the true aspect ratio. For example, due to perspective distortion, the ID card area is compressed into a square after cropping and transformation, which cannot reflect the true aspect ratio. The fonts and patterns inside are also compressed. Therefore, it is necessary to calculate the true aspect ratio of the element area and restore the ratio.

[0004] In the related art, the average length and average width of the element region are calculated as the real aspect ratio of the element region, and the element region is restored to proportion based on this. Due to perspective distortion, the average length and average width of the element region cannot reflect the real aspect ratio of the element region, and restoring the element region to proportion based on this may easily cause the proportion of the restored element region to be distorted. Summary of the invention

[0005] In order to solve the technical problem that due to perspective distortion, the average length and average width of the element area cannot reflect the true aspect ratio of the element area, and the proportional restoration of the element area is easily distorted, the embodiment of the present application provides an image processing method, device, electronic device and storage medium. The specific technical solution is as follows:

[0006] In a first aspect of an embodiment of the present application, an image processing method is first provided, the method comprising:

[0007] Acquire an original image, and determine an element region of a target element in the original image;

[0008] Acquire geometric parameters of the original image and vertex coordinates of the element area;

[0009] Determining the restored geometric proportion of the element region according to the vertex coordinates and the geometric parameters;

[0010] Based on the restored geometric proportion, the element region is restored in proportion to obtain a target element region.

[0011] In an optional implementation, determining the restored geometric proportion of the element region according to the vertex coordinates and the geometric parameters includes:

[0012] A binary image corresponding to the original image is determined, and a restored geometric proportion of the element region is determined according to the binary image, the vertex coordinates and the geometric parameters.

[0013] In an optional implementation, determining the binary image corresponding to the original image includes:

[0014] Obtaining a preliminary binary image corresponding to the original image, scaling the preliminary binary image corresponding to the original image in equal proportion to obtain a scaled binary image, wherein the length of the scaled binary image reaches a preset threshold;

[0015] A preset third pixel value is filled in the scaled binary image to obtain the binary image, wherein the length and width of the binary image both reach the preset threshold.

[0016] In an optional implementation, the obtaining of a preliminary binarized image corresponding to the original image includes:

[0017] The element area is filled with a preset first pixel value, and the non-element area in the original image is filled with a preset second pixel value, so as to obtain a preliminary binarized image corresponding to the original image.

[0018] In an optional implementation, determining the restored geometric proportion of the element region according to the binary image, the vertex coordinates and the geometric parameters includes:

[0019] The vertex coordinates and the geometric parameters are spliced ​​into a one-dimensional vector, and the restored geometric proportion of the element region is determined according to the one-dimensional vector and the binarized image.

[0020] In an optional implementation, determining the restored geometric proportion of the element region according to the one-dimensional vector and the binarized image includes:

[0021] The one-dimensional vector and the binarized image are input into a pre-trained geometric proportion prediction network to obtain the restored geometric proportion of the element region.

[0022] In an optional embodiment, the pre-trained geometric proportion prediction network includes a pre-trained convolutional neural network, a pre-trained first multi-layer perceptron, and a pre-trained second multi-layer perceptron;

[0023] The step of inputting the one-dimensional vector and the binarized image into a pre-trained geometric proportion prediction network to obtain the restored geometric proportion of the element region includes:

[0024] Inputting the binarized image into the pre-trained convolutional neural network to obtain an image vector;

[0025] Inputting the one-dimensional vector into the pre-trained first multi-layer perceptron to obtain a geometric vector;

[0026] Splicing the image vector with the geometric vector to obtain a spliced ​​vector;

[0027] The splicing vector is input into the pre-trained second multi-layer perceptron to obtain the restored geometric proportion of the element area.

[0028] In an optional embodiment, the restoring geometric proportion is restoring the aspect ratio;

[0029] The method of performing proportional restoration on the element region based on the restored geometric proportion to obtain a target element region includes:

[0030] Determining a target length and a target width of the element region according to the restored aspect ratio;

[0031] Constructing restored vertex coordinates of the element area according to the target length and the target width;

[0032] According to the vertex coordinates and the restored vertex coordinates, the element region is proportionally restored to obtain a target element region;

[0033] The method of performing proportional restoration on the element region according to the vertex coordinates and the restored vertex coordinates to obtain a target element region includes:

[0034] Determine a perspective transformation matrix according to the vertex coordinates and the restored vertex coordinates;

[0035] According to the perspective transformation matrix, the element region is perspective transformed to obtain a target element region.

[0036] In an optional implementation, determining the target length and target width of the element region according to the restored aspect ratio includes:

[0037] Determining the original aspect ratio of the element region, and comparing the restored aspect ratio with the original aspect ratio;

[0038] According to the comparison result, the target length and target width of the element area are determined.

[0039] In an optional implementation, determining the original aspect ratio of the element region includes:

[0040] Obtaining a maximum length and a maximum width of the element region, and determining a first ratio between the maximum length and the maximum width;

[0041] The first ratio is determined as the original aspect ratio of the element region.

[0042] In an optional implementation, determining the target length and target width of the element region according to the comparison result includes:

[0043] In a case where the original aspect ratio is greater than or equal to the restored aspect ratio, determining the maximum length of the element region as the target length of the element region; determining a second ratio between the target length and the restored aspect ratio, and determining the second ratio as the target width of the element region;

[0044] Alternatively, when the original aspect ratio is smaller than the restored aspect ratio, the maximum width of the element region is determined as the target width of the element region; the product between the target width and the restored aspect ratio is determined, and the product is determined as the target length of the element region.

[0045] In an optional implementation, determining a perspective transformation matrix according to the vertex coordinates and the restored vertex coordinates includes:

[0046] The vertex coordinates and the restored vertex coordinates are input into a perspective transformation formula, and the perspective transformation formula is solved to obtain a perspective transformation matrix.

[0047] In an optional implementation, performing perspective transformation on the element region according to the perspective transformation matrix to obtain the target element region includes:

[0048] For any pixel in the element area, inputting the pixel coordinates of the pixel into a perspective transformation formula to obtain the restored coordinates corresponding to the pixel;

[0049] The perspective transformation formula is:

[0050] [x, y, c] = [u, v, c]*M;

[0051] Wherein, x and y are pixel coordinates of the pixel, u and v are restored coordinates, M is a perspective transformation matrix, and c is a constant;

[0052] A target element region is formed according to the restored coordinates of any pixel in the element region.

[0053] In an optional embodiment, the pre-trained geometric ratio prediction network is obtained by:

[0054] Acquire a sample image and a true geometric ratio of a sample element in the sample image, and determine a sample element area of ​​the sample element in the sample image;

[0055] Acquire sample geometric parameters of the sample image and sample vertex coordinates of the sample element area;

[0056] Determine a sample binary image corresponding to the sample image, and concatenate the sample vertex coordinates and the sample geometric parameters into a sample one-dimensional vector;

[0057] Inputting the sample one-dimensional vector and the sample binary image into a geometric ratio prediction network to obtain a predicted geometric ratio of the sample element region;

[0058] Determining a proportional loss between the predicted geometric proportion and the true geometric proportion, and training the geometric proportion prediction network based on the proportional loss;

[0059] When the scale loss converges, the training is stopped to obtain a pre-trained geometric scale prediction network.

[0060] In a second aspect of the embodiments of the present application, an image processing device is further provided, the device comprising:

[0061] An image acquisition module, used to acquire an original image and determine an element region of a target element in the original image;

[0062] A parameter and coordinate acquisition module, used for acquiring the geometric parameters of the original image and the vertex coordinates of the element area;

[0063] A proportion determination module, used for determining the restored geometric proportion of the element area according to the vertex coordinates and the geometric parameters;

[0064] The region restoration module is used to restore the element region in proportion based on the restored geometric proportion to obtain a target element region.

[0065] In an optional implementation, the ratio determination module specifically includes:

[0066] A binarized image determination submodule, used to determine a binarized image corresponding to the original image;

[0067] The proportion determination submodule is used to determine the restored geometric proportion of the element area according to the binary image, the vertex coordinates and the geometric parameters.

[0068] In an optional implementation, the binarized image determination submodule specifically includes:

[0069] A binary image acquisition unit, used to acquire a preliminary binary image corresponding to the original image;

[0070] A binary image scaling unit is used to scale the preliminary binary image corresponding to the original image in equal proportion to obtain a scaled binary image, wherein the length of the scaled binary image reaches a preset threshold;

[0071] The binary image filling unit is used to fill a preset third pixel value in the scaled binary image to obtain the binary image, and the length and width of the binary image both reach the preset threshold.

[0072] In an optional implementation, the binary image acquisition unit is specifically used to:

[0073] The element area is filled with a preset first pixel value, and the non-element area in the original image is filled with a preset second pixel value, so as to obtain a binary image corresponding to the original image.

[0074] In an optional implementation, the ratio determination submodule specifically includes:

[0075] A coordinate and parameter splicing unit, used for splicing the vertex coordinates and the geometric parameters into a one-dimensional vector;

[0076] A ratio determination unit is used to determine the restored geometric ratio of the element area according to the one-dimensional vector and the binarized image.

[0077] In an optional implementation, the ratio determination unit is specifically configured to:

[0078] The one-dimensional vector and the binarized image are input into a pre-trained geometric proportion prediction network to obtain the restored geometric proportion of the element region.

[0079] In an optional embodiment, the pre-trained geometric proportion prediction network includes a pre-trained convolutional neural network, a pre-trained first multi-layer perceptron, and a pre-trained second multi-layer perceptron; the proportion determination unit is specifically used for:

[0080] Inputting the binarized image into the pre-trained convolutional neural network to obtain an image vector;

[0081] Inputting the one-dimensional vector into the pre-trained first multi-layer perceptron to obtain a geometric vector;

[0082] Splicing the image vector with the geometric vector to obtain a spliced ​​vector;

[0083] The splicing vector is input into the pre-trained second multi-layer perceptron to obtain the restored geometric proportion of the element area.

[0084] In an optional implementation, the geometric ratio restoration is to restore the length-width ratio; the region restoration module specifically includes:

[0085] A length-width determination submodule, used to determine a target length and a target width of the element region according to the restored length-width ratio;

[0086] A coordinate construction submodule, used for constructing the restored vertex coordinates of the element area according to the target length and the target width;

[0087] A region transformation submodule, used for performing proportional restoration on the element region according to the vertex coordinates and the restored vertex coordinates to obtain a target element region;

[0088] The area transformation submodule specifically includes:

[0089] A matrix determination unit, used for determining a perspective transformation matrix according to the vertex coordinates and the restored vertex coordinates;

[0090] The region transformation unit is used to perform perspective transformation on the element region according to the perspective transformation matrix to obtain a target element region.

[0091] In an optional implementation, the length and width determination submodule specifically includes:

[0092] an original aspect ratio determining unit, used to determine the original aspect ratio of the element region;

[0093] A ratio comparison unit, used to compare the restored aspect ratio with the original aspect ratio;

[0094] The length and width determination unit is used to determine the target length and target width of the element area according to the comparison result.

[0095] In an optional implementation, the original aspect ratio determination unit is specifically configured to:

[0096] Obtaining a maximum length and a maximum width of the element region, and determining a first ratio between the maximum length and the maximum width;

[0097] The first ratio is determined as the original aspect ratio of the element region.

[0098] In an optional implementation, the length and width determination unit is specifically configured to:

[0099] In a case where the original aspect ratio is greater than or equal to the restored aspect ratio, determining the maximum length of the element region as the target length of the element region; determining a second ratio between the target length and the restored aspect ratio, and determining the second ratio as the target width of the element region;

[0100] Alternatively, when the original aspect ratio is smaller than the restored aspect ratio, the maximum width of the element region is determined as the target width of the element region; the product between the target width and the restored aspect ratio is determined, and the product is determined as the target length of the element region.

[0101] In an optional implementation, the matrix determination unit is specifically configured to:

[0102] The vertex coordinates and the restored vertex coordinates are input into a perspective transformation formula, and the perspective transformation formula is solved to obtain a perspective transformation matrix.

[0103] In an optional implementation, the region transformation unit is specifically configured to:

[0104] For any pixel in the element area, inputting the pixel coordinates of the pixel into a perspective transformation formula to obtain the restored coordinates corresponding to the pixel;

[0105] The perspective transformation formula is:

[0106] [x, y, c] = [u, v, c]*M;

[0107] Wherein, x and y are pixel coordinates of the pixel, u and v are restored coordinates, M is a perspective transformation matrix, and c is a constant;

[0108] A target element region is formed according to the restored coordinates of any pixel in the element region.

[0109] In an optional embodiment, the device further includes: a network training module, specifically configured to:

[0110] Acquire a sample image and a true geometric ratio of a sample element in the sample image, and determine a sample element area of ​​the sample element in the sample image;

[0111] Acquire sample geometric parameters of the sample image and sample vertex coordinates of the sample element area;

[0112] Determine a sample binary image corresponding to the sample image, and concatenate the sample vertex coordinates and the sample geometric parameters into a sample one-dimensional vector;

[0113] Inputting the sample one-dimensional vector and the sample binary image into a geometric ratio prediction network to obtain a predicted geometric ratio of the sample element region;

[0114] Determining a proportional loss between the predicted geometric proportion and the true geometric proportion, and training the geometric proportion prediction network based on the proportional loss;

[0115] When the scale loss converges, the training is stopped to obtain a pre-trained geometric scale prediction network.

[0116] In a third aspect of the embodiments of the present application, there is further provided an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0117] Memory, used to store computer programs;

[0118] The processor is used to implement any image processing method described in the first aspect when executing the program stored in the memory.

[0119] In a fourth aspect of the embodiments of the present application, a storage medium is further provided, wherein instructions are stored in the storage medium, and when the storage medium is run on a computer, the computer executes the image processing method described in any one of the first aspects above.

[0120] In a fifth aspect of the embodiments of the present application, a computer program product comprising instructions is also provided, which, when executed on a computer, enables the computer to execute any of the above-mentioned image processing methods.

[0121] The technical solution provided in the embodiment of the present application obtains the original image and determines the element area of ​​the target element in the original image, obtains the geometric parameters of the original image and the vertex coordinates of the element area, determines the restored geometric proportion of the element area according to the vertex coordinates and the geometric parameters, and restores the proportion of the element area based on the restored geometric proportion to obtain the target element area.

[0122] In this way, based on the geometric parameters of the original image and the vertex coordinates of the element area, the restored geometric proportion of the element area is determined, and the proportion of the element area is restored accordingly, thereby avoiding distortion of the proportion of the restored element area. BRIEF DESCRIPTION OF THE DRAWINGS

[0123] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0124] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0125] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0126] Figure 1 A schematic diagram of an implementation flow of an image processing method shown in an embodiment of the present application;

[0127] Figure 2 A schematic diagram of a document image shown in an embodiment of the present application;

[0128] Figure 3 It is a schematic diagram of an implementation flow of another image processing method shown in an embodiment of the present application;

[0129] Figure 4 A schematic diagram of the architecture of a pre-trained geometric ratio prediction network shown in an embodiment of the present application;

[0130] Figure 5 A schematic diagram of an implementation flow of a region transformation method shown in an embodiment of the present application;

[0131] Figure 6 This is a schematic diagram of a document area after restoration shown in an embodiment of the present application;

[0132] Figure 7 A schematic diagram of an implementation flow of a network training method shown in an embodiment of the present application;

[0133] Figure 8 A schematic diagram of the structure of an image processing device shown in an embodiment of the present application;

[0134] Fig. 9 It is a schematic diagram of the structure of an electronic device shown in an embodiment of the present application. DETAILED DESCRIPTION

[0135] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0136] The disclosure below provides many different embodiments or examples to realize the different structures of the present application. In order to simplify the disclosure of the present application, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present application. In addition, the present application can repeat reference numbers and / or letters in different examples. This repetition is for the purpose of simplification and clarity, and does not itself indicate the relationship between the various embodiments and / or settings discussed.

[0137] like Figure 1 FIG. 1 is a schematic diagram of an implementation flow of an image processing method provided in an embodiment of the present application. The method is applied to an electronic device and may specifically include the following steps:

[0138] S101, acquiring an original image, and determining an element region of a target element in the original image.

[0139] In the embodiment of the present application, an original image is obtained, wherein the original image may be, for example, a document image obtained by photographing a document, an ID image obtained by photographing an ID, or a screen image obtained by photographing a computer screen, which is not limited in the embodiment of the present application.

[0140] For the original image, the element area of ​​the target element in the original image is determined. The target element may be, for example, a document, a certificate, or a computer screen, which is not limited in the embodiment of the present application.

[0141] For example, to obtain a document image as a raw image, such as Figure 2 As shown, the document area of ​​the document in the original image is determined. The document area can be determined by using an existing document area detection method, which is not limited in the present embodiment.

[0142] S102, obtaining geometric parameters of the original image and vertex coordinates of the element area.

[0143] In the embodiment of the present application, the geometric parameters of the original image and the vertex coordinates of the element area of ​​the target element in the original image are obtained. The geometric parameters of the original image generally refer to the length and width of the original image.

[0144] It should be noted that the vertex coordinates of the element area of ​​the target element in the original image usually refer to the coordinates of four vertices, namely the upper left vertex coordinates, the lower left vertex coordinates, the upper right vertex coordinates, and the lower right vertex coordinates. The embodiment of the present application does not limit this.

[0145] For example, obtain the length L and width W of the original image, as well as the upper left vertex coordinates (x1, y1), lower left vertex coordinates (x2, y2), upper right vertex coordinates (x3, y3), and lower right vertex coordinates (x4, y4) of the element area of ​​the target element in the original image.

[0146] S103, determining the restored geometric proportion of the element area according to the vertex coordinates and the geometric parameters.

[0147] In the embodiment of the present application, according to the imaging principle, the vertex coordinates of the element area of ​​the target element in the original image and the geometric parameters of the original image need to be known in order to calculate the restored geometric proportions of the element area of ​​the target element in the original image.

[0148] Based on this, the above steps can obtain the geometric parameters of the original image and the vertex coordinates of the element area of ​​the target element in the original image, and thus the restored geometric proportion of the element area can be determined according to the vertex coordinates and the geometric parameters.

[0149] S104, based on the restored geometric proportion, the element region is restored in proportion to obtain a target element region.

[0150] In an embodiment of the present application, after the above steps, the restored geometric proportion of the element area can be obtained, and then based on the restored geometric proportion, the element area is proportionally restored to obtain the target element area, which can be subsequently displayed or shared.

[0151] In this way, referring to the imaging principle, based on the geometric parameters of the original image and the vertex coordinates of the element area, the restored geometric proportion of the element area is determined, and the proportion of the element area is restored accordingly, which can avoid the distortion of the proportion of the restored element area.

[0152] In addition, in the embodiment of the present application, the geometric parameters of the original image and the vertex coordinates of the element region contain limited information, and it is difficult to extract rich information, and it is difficult to calculate the restored geometric proportion of the element region. However, the binary image corresponding to the original image can provide more information, so the binary image corresponding to the original image can be added on the basis of the geometric parameters of the original image and the vertex coordinates of the element region to jointly calculate the restored geometric proportion of the element region.

[0153] Based on this, Figure 3 FIG. 1 is a schematic diagram of an implementation flow of another image processing method provided in an embodiment of the present application. The method is applied to an electronic device and may specifically include the following steps:

[0154] S301, acquiring an original image, and determining an element region of a target element in the original image.

[0155] In the embodiment of the present application, this step is similar to the above-mentioned step S101, and the embodiment of the present application will not be described one by one here.

[0156] S302, obtaining geometric parameters of the original image and vertex coordinates of the element area.

[0157] In the embodiment of the present application, this step is similar to the above-mentioned step S102, and the embodiment of the present application will not be described one by one here.

[0158] S303, determining a binary image corresponding to the original image, and determining a restored geometric proportion of the element region according to the binary image, vertex coordinates and geometric parameters.

[0159] In an embodiment of the present application, the geometric parameters of the original image and the vertex coordinates of the element area of ​​the target element in the original image can be obtained through the above steps. In addition, it is necessary to determine the binary image corresponding to the original image, thereby determining the restored geometric proportions of the element area based on the binary image, vertex coordinates and geometric parameters.

[0160] For determining the binary image: obtaining a preliminary binary image corresponding to the original image, scaling the preliminary binary image corresponding to the original image to obtain a scaled binary image, the length of the scaled binary image reaches a preset threshold, filling a preset third pixel value in the scaled binary image to obtain a binary image, the length and width of the binary image both reach the preset threshold.

[0161] Among them, for the preliminary binary image corresponding to the original image, the element area of ​​the target element in the original image can usually be segmented to obtain it. Specifically, in one embodiment of the present application, the element area of ​​the target element in the original image can be filled with a preset first pixel value (for example, 0), and the non-element area in the original image can be filled with a preset second pixel value (for example, 255), so as to obtain the preliminary binary image corresponding to the original image. In another embodiment of the present application, other methods can also be used to achieve the segmentation of the element area and the non-element area of ​​the target element in the original image, such as using other different colors to mark the document area and the non-document area. The element areas are identified in sequence.

[0162] In addition, in order to retain the original information of the element area (such as the document area), the preliminary binary image needs to be scaled to a size of, for example, 512 on the longest side, the other side is scaled proportionally, and then padded to an image size of, for example, 512×512.

[0163] Specifically, the preliminary binary image is scaled proportionally to obtain a scaled binary image, wherein the length of the scaled binary image reaches a preset threshold value (e.g., 512), and a preset third pixel value (e.g., 128) is filled in the scaled binary image to obtain a binary image, wherein both the length and width of the binary image reach a preset threshold value (e.g., 512), which means that for the scaled binary image, its width needs to be filled to the preset threshold value (e.g., 512).

[0164] Thus, for the preliminary binary image, a binary image is obtained through proportional scaling, pixel filling and other processes, and its final size is, for example, 512×512. Thus, the restored geometric proportion of the element area is determined according to the binary image, vertex coordinates and geometric parameters.

[0165] The geometric parameters of the original image and the vertex coordinates of the element region of the target element in the original image can be spliced ​​into a one-dimensional vector, thereby determining the restored geometric proportion of the element region based on the one-dimensional vector and the binarized image.

[0166] For example, the geometric parameters of the original image are length L and width W, and the vertex coordinates of the element area of ​​the target element in the original image are the upper left vertex coordinate (x1, y1), the lower left vertex coordinate (x2, y2), the upper right vertex coordinate (x3, y3), and the lower right vertex coordinate (x4, y4), which are spliced ​​into a one-dimensional vector [x1, y1, x2, y2, x3, y3, x4, y4, L, W]. According to the one-dimensional vector and the binary image, the restored geometric proportion of the element area is determined.

[0167] In an embodiment of the present application, a pre-trained geometric proportion prediction network is provided, and the above-mentioned one-dimensional vector and the binarized image can be input into the pre-trained geometric proportion prediction network to obtain the restored geometric proportion of the element area.

[0168] It should be noted that the geometric parameters of the original image and the vertex coordinates of the element area of ​​the target element in the original image are concatenated into a one-dimensional vector as one of the inputs of the pre-trained geometric ratio prediction network. However, the one-dimensional vector contains limited information, and the pre-trained geometric ratio prediction network is difficult to extract rich information. It is difficult to calculate the restored geometric ratio of the element area, and the binary image can provide more information. Therefore, the binary image is used as another input of the pre-trained geometric ratio prediction network.

[0169] The pre-trained geometric ratio prediction network mainly includes a pre-trained convolutional neural network, a pre-trained first multi-layer perceptron, and a pre-trained second multi-layer perceptron. The structures of the pre-trained first multi-layer perceptron and the pre-trained second multi-layer perceptron may be the same or different.

[0170] Therefore, the binarized image is input into a pre-trained convolutional neural network to obtain an image vector, the one-dimensional vector is input into a pre-trained first multi-layer perceptron to obtain a geometric vector, the image vector and the geometric vector are spliced ​​to obtain a spliced ​​vector, and the spliced ​​vector is input into a pre-trained second multi-layer perceptron to obtain the restored geometric proportion of the element area.

[0171] For example, Figure 4 As shown, the binarized image is input into the pre-trained convolutional neural network to obtain a 1×512 image vector, the one-dimensional vector [x1, y1, x2, y2, x3, y3, x4, y4, L, W] is input into the pre-trained first multi-layer perceptron to obtain a 1×512 geometric vector, the image vector and the geometric vector are spliced ​​to obtain a 1×1024 spliced ​​vector, and the spliced ​​vector is input into the pre-trained second multi-layer perceptron to obtain the restored geometric ratio pred_ratio of the element area.

[0172] It should be noted that, for the pre-trained convolutional neural network, its architecture can be similar to the architecture of the existing convolutional neural network, and the same is true for the pre-trained first multi-layer perceptron and the pre-trained second multi-layer perceptron. The embodiments of the present application are not limited to this.

[0173] S304, based on the restored geometric proportion, the element region is restored in proportion to obtain a target element region.

[0174] In the embodiment of the present application, after the above steps, the restored geometric ratio of the element area of ​​the target element in the original image can be obtained, and thus the element area can be proportionally restored according to the restored geometric ratio to obtain the target element area.

[0175] The restored geometric ratio of the element region of the target element in the original image usually refers to the restored aspect ratio, for example, the restored aspect ratio is 2:1, and thus the element region is proportionally restored based on the restored aspect ratio to obtain the target element region.

[0176] For the process of restoring the proportion of the element area based on restoring the aspect ratio, please refer to the following: Figure 5 As shown in the method. Figure 5 FIG. 1 is a schematic diagram of an implementation flow of a region conversion method provided in an embodiment of the present application. The method is applied to an electronic device and may specifically include the following steps:

[0177] S501, determining a target length and a target width of the element area according to the restored aspect ratio.

[0178] In the embodiment of the present application, for the restored aspect ratio of the element region of the target element in the original image, the target length and target width of the element region of the target element in the original image may be determined according to the restored aspect ratio.

[0179] Among them, for the element area of ​​the target element in the original image, the original aspect ratio of the element area is determined, the original aspect ratio is compared with the restored aspect ratio to obtain a comparison result, and according to the comparison result, the target length and target width of the element area of ​​the target element in the original image are determined.

[0180] For the element region of the target element in the original image, which is usually a quadrilateral, the maximum length of the upper and lower long sides and the maximum width of the left and right wide sides can be obtained, and the original length-width ratio of the element region is calculated accordingly. Specifically, the maximum length and the maximum width of the element region are obtained, and a first ratio between the maximum length and the maximum width is determined, and the first ratio is determined as the original length-width ratio of the element region.

[0181] For example, the initialRatio of the element area is as follows:

[0182] initialRatio=maxLength / maxWidth.

[0183] Among them, maxLength is the maximum length of the upper and lower long sides of the element area, and maxWidth is the maximum width of the left and right wide sides of the element area.

[0184] For the comparison result of the original aspect ratio and the restored aspect ratio, when the original aspect ratio is greater than or equal to the restored aspect ratio, the maximum length of the element area is determined as the target length of the element area, a second ratio between the target length and the restored aspect ratio is determined, and the second ratio is determined as the target width of the element area; or, when the original aspect ratio is less than the restored aspect ratio, the maximum width of the element area is determined as the target width of the element area, the product between the target width and the restored aspect ratio is determined, and the product is determined as the target length of the element area.

[0185] For example, the original aspect ratio initialRatio and the restored aspect ratio pred_ratio, when the original aspect ratio initialRatio ≥ the restored aspect ratio pred_ratio, the calculated target length L and target width W of the element area are as follows:

[0186] L = maxLength;

[0187] W = L / pred_ratio;

[0188] When the original aspect ratio initialRatio is less than the restored aspect ratio pred_ratio, the target length L and target width W of the element area are calculated as follows:

[0189] W = maxWidth;

[0190] L = W*pred_ratio;

[0191] Among them, maxLength is the maximum length of the upper and lower long sides of the element area, and maxWidth is the maximum width of the left and right wide sides of the element area.

[0192] S502: construct restored vertex coordinates of the element area according to the target length and the target width.

[0193] In the embodiment of the present application, after the above steps, the target length and target width of the element area can be obtained, and thus the restored vertex coordinates of the element area can be constructed according to the target length and target width.

[0194] The restored vertex coordinates of the element region are usually the original lower left vertex coordinates, the original upper left vertex coordinates, the original upper right vertex coordinates, and the original lower right vertex coordinates. For example, the target length L and the target width W of the element region can construct the original lower left vertex coordinates (0, 0), the original upper left vertex coordinates (0, W), the original upper right vertex coordinates (L, W), and the original lower right vertex coordinates (W, 0) of the element region.

[0195] S503, performing proportional restoration on the element region according to the vertex coordinates and the restored vertex coordinates to obtain a target element region.

[0196] In the embodiment of the present application, the vertex coordinates and restored vertex coordinates of the element region can be obtained through the above steps. According to the vertex coordinates and the restored vertex coordinates, the element region is proportionally restored to obtain the target element region.

[0197] Specifically, in one embodiment of the present application, the element region is proportionally restored according to the vertex coordinates and the restored vertex coordinates to obtain the target element region, including: determining the perspective transformation matrix according to the vertex coordinates and the restored vertex coordinates, and performing perspective transformation on the element region according to the perspective transformation matrix to obtain the target element region. In another embodiment of the present application, the element region can also be proportionally restored according to the vertex coordinates and the restored vertex coordinates by using other comparison transformation methods to obtain the target element region. Among them, the other comparison transformation method can reflect the comparison relationship between the vertex coordinates and the restored vertex coordinates.

[0198] Among them, the perspective transformation matrix is ​​determined according to the vertex coordinates and the restored vertex coordinates. According to the perspective transformation matrix, the element area is perspective transformed. When the target element area is obtained, a perspective transformation formula can be provided, and the vertex coordinates of the element area and the restored vertex coordinates are input into the perspective transformation formula. Then, the perspective transformation formula is solved to obtain the perspective transformation matrix.

[0199] For example, the vertex coordinates of the element area are the upper left vertex coordinates (x1, y1), the lower left vertex coordinates (x2, y2), the upper right vertex coordinates (x3, y3), and the lower right vertex coordinates (x4, y4), and the corresponding restored vertex coordinates are the original lower left vertex coordinates (0, 0), the original upper left vertex coordinates (0, W), the original upper right vertex coordinates (L, W), and the original lower right vertex coordinates (W, 0). Then the upper left vertex coordinates (x1, y1) and the original upper left vertex coordinates (0, W) are input into the following perspective transformation formula, and the lower left vertex coordinates (x2, y2) and the original lower left vertex coordinates (0, 0) are input into the following perspective transformation formula, and so on. Then, 4 sets of perspective transformation formulas to be solved are obtained, and the 4 sets of perspective transformation formulas to be solved are solved to obtain the perspective transformation matrix.

[0200]

[0201] Among them, c is a constant, for example, it can be 1, M is the perspective transformation matrix, x and y are the vertex coordinates, and u and v are the restored vertex coordinates corresponding to the vertex coordinates.

[0202] In addition, according to the perspective transformation matrix, the element region is perspective transformed to obtain the target element region. Specifically, according to the perspective transformation matrix, the pixel coordinates of each pixel in the element region are perspective transformed to obtain the target element region.

[0203] Specifically, for any pixel in the element area, the pixel coordinates of the pixel are input into the perspective transformation formula to obtain the restored coordinates corresponding to the pixel;

[0204] The perspective transformation formula is:

[0205] [x, y, c] = [u, v, c]*M;

[0206] Wherein, x and y are pixel coordinates of the pixel, u and v are restored coordinates, M is the perspective transformation matrix, and c is a constant, which may be 1, for example;

[0207] According to the restored coordinates of any pixel in the element area, the target element area is formed, such as Figure 6 shown.

[0208] In this way, based on the geometric parameters of the original image, the vertex coordinates of the element area and the binary image corresponding to the original image, the restored geometric proportion of the element area is determined, and the proportion of the element area is restored accordingly, thereby avoiding distortion of the proportion of the restored element area.

[0209] In addition, in the embodiment of the present application, a schematic diagram of the implementation flow of a network training method is also provided, such as Figure 7 As shown, the method is applied to an electronic device and may specifically include the following steps:

[0210] S701, obtaining a sample image and a real geometric ratio of a sample element in the sample image, and determining a sample element region of the sample element in the sample image.

[0211] In the embodiment of the present application, a sample image is obtained, wherein the sample image may be, for example, a document image obtained by photographing a document, an ID image obtained by photographing an ID, or a screen image obtained by photographing a computer screen, which is not limited in the embodiment of the present application.

[0212] In addition, it is also necessary to obtain the real geometric proportion of the sample element in the sample image. The sample element may be, for example, a document, a certificate, or a computer screen, which is not limited in the present embodiment.

[0213] In addition, it is also necessary to determine the sample element region of the sample element in the sample image. The sample element region of the sample element in the sample image can be determined by using an existing detection method such as a document region, which is not limited in the present embodiment.

[0214] S702, obtaining sample geometric parameters of the sample image and sample vertex coordinates of the sample element area.

[0215] In the embodiment of the present application, the sample geometric parameters of the sample image and the sample vertex coordinates of the sample element area of ​​the sample element in the sample image are obtained. The sample geometric parameters of the sample image generally refer to the length and width of the sample image.

[0216] It should be noted that the sample vertex coordinates of the sample element area of ​​the sample element in the sample image usually refer to the coordinates of four vertices, namely the upper left vertex coordinates, the lower left vertex coordinates, the upper right vertex coordinates, and the lower right vertex coordinates, which are not limited in this embodiment of the present application.

[0217] S703, determining a sample binary image corresponding to the sample image, and concatenating the sample vertex coordinates and the sample geometric parameters into a sample one-dimensional vector.

[0218] In the embodiment of the present application, the sample geometric parameters of the sample image and the sample vertex coordinates of the sample element area of ​​the sample element in the sample image can be obtained through the above steps. In addition, the sample binary image corresponding to the sample image needs to be determined.

[0219] For determining the sample binary image: obtain a sample preliminary binary image corresponding to the sample image, scale the sample preliminary binary image proportionally to obtain a sample scaled binary image, the length of the sample scaled binary image reaches a preset threshold, fill a preset third pixel value in the sample scaled binary image to obtain a sample binary image, and both the length and width of the sample binary image reach the preset threshold.

[0220] Among them, for the sample preliminary binary image corresponding to the sample image, the sample element area of ​​the sample element in the sample image can usually be segmented to obtain it. Specifically, in one embodiment of the present application, the sample element area of ​​the sample element in the sample image can be filled with a preset first pixel value (for example, 0), and the non-sample element area in the sample image can be filled with a preset second pixel value (for example, 255), so as to obtain the sample preliminary binary image corresponding to the sample image. In another embodiment of the present application, other methods can also be used to achieve the segmentation of the sample element area and the non-sample element area of ​​the sample element in the sample image, such as using other different colors to mark the document area and the non-document area, and identifying the element areas in turn.

[0221] In addition, in order to preserve the original information of the sample element area, the sample preliminary binary image needs to be scaled to a size of, for example, 512 according to the longest side, and the other side is scaled in proportion, and then padded to, for example, an image size of 512×512.

[0222] Specifically, the sample preliminary binary image is scaled proportionally to obtain a sample scaled binary image, wherein the length of the sample scaled binary image reaches a preset threshold value (e.g., 512), and a preset third pixel value (e.g., 128) is filled in the sample scaled binary image to obtain a sample binary image, wherein both the length and width of the sample binary image reach a preset threshold value (e.g., 512), which means that for the sample scaled binary image, its width is filled to the preset threshold value (e.g., 512). In this way, for the sample binary image, after proportional scaling, pixel filling and other processes, the sample binary image is obtained, and its final size is, for example, 512×512.

[0223] In addition, the sample geometric parameters of the sample image and the sample vertex coordinates of the sample element region of the sample element in the sample image can be spliced ​​into a sample one-dimensional vector. For example, the sample geometric parameters of the sample image are length L and width W, and the sample vertex coordinates of the sample element region of the sample element in the sample image are the upper left vertex coordinate (x1, y1), the lower left vertex coordinate (x2, y2), the upper right vertex coordinate (x3, y3), and the lower right vertex coordinate (x4, y4), which are spliced ​​into a sample one-dimensional vector [x1, y1, x2, y2, x3, y3, x4, y4, L, W].

[0224] S704, inputting the sample one-dimensional vector and the sample binary image into a geometric ratio prediction network to obtain a predicted geometric ratio of the sample element region.

[0225] In an embodiment of the present application, the sample one-dimensional vector obtained in the above steps and the sample binary image corresponding to the sample image can be input into a geometric proportion prediction network to obtain the predicted geometric proportion of the sample element area.

[0226] The geometric ratio prediction network mainly includes a convolutional neural network, a first multilayer perceptron and a second multilayer perceptron. Among them, the structures of the first multilayer perceptron and the second multilayer perceptron can be the same or different. Thus, the sample one-dimensional vector is input into the first multilayer perceptron to obtain a predicted geometric vector, the sample binary image is input into the convolutional neural network to obtain a predicted image vector, the predicted image vector is spliced ​​with the predicted geometric vector to obtain a predicted splicing vector, and the predicted splicing vector is input into the second multilayer perceptron to obtain the predicted geometric ratio of the sample element area. By splicing the predicted image vector with the predicted geometric vector, the model can extract rich information and make the prediction more accurate. In addition, the sample binary image is scaled to a size of 512 according to the longest side, the other side is scaled proportionally, and then filled to an image size of 512×512, which retains more original information of the quadrilateral and makes the predicted geometric ratio more accurate.

[0227] S705, determining the proportional loss between the predicted geometric proportion and the actual geometric proportion, and training the geometric proportion prediction network according to the proportional loss.

[0228] S706, when the scale loss converges, stop training to obtain a pre-trained geometric scale prediction network.

[0229] In an embodiment of the present application, the proportional loss between the predicted geometric proportion and the actual geometric proportion is determined, and the geometric proportion prediction network is trained based on the proportional loss. When the proportional loss converges, for example, the proportional loss is less than a certain threshold, the training is stopped to obtain a pre-trained geometric proportion prediction network.

[0230] Among them, the determination of the ratio loss between the predicted geometric ratio and the actual geometric ratio can be determined by using, for example, the MSE loss function, as shown below, and the embodiment of the present application does not limit this.

[0231] Loss=(pred_ratio-real_ratio) 2 .

[0232] Among them, Loss is the ratio loss, and real_ratio is the real geometric ratio, for example pred_ratio is the predicted geometric ratio.

[0233] After the above steps, a pre-trained geometric proportion prediction network can be obtained. Subsequently, the geometric parameters of the original image, the vertex coordinates of the element area and the binary image corresponding to the original image are input into the pre-trained geometric proportion prediction network to obtain the restored geometric proportion of the element area. The proportion of the element area is restored based on this, which can avoid distortion of the proportion of the restored element area.

[0234] Corresponding to the above method embodiment, the present application embodiment also provides an image processing device, such as Figure 8 As shown, the device may include: an image acquisition module 810 , a parameter and coordinate acquisition module 820 , a ratio determination module 830 , and a region restoration module 840 .

[0235] The image acquisition module 810 is used to acquire an original image and determine an element region of a target element in the original image;

[0236] A parameter and coordinate acquisition module 820, used to acquire the geometric parameters of the original image and the vertex coordinates of the element area;

[0237] A ratio determination module 830, configured to determine a restored geometric ratio of the element region according to the vertex coordinates and the geometric parameters;

[0238] The region restoration module 840 is used to restore the proportion of the element region based on the restored geometric proportion to obtain a target element region.

[0239] In an optional implementation, the ratio determination module specifically includes:

[0240] A binarized image determination submodule, used to determine a binarized image corresponding to the original image;

[0241] The proportion determination submodule is used to determine the restored geometric proportion of the element area according to the binary image, the vertex coordinates and the geometric parameters.

[0242] In an optional implementation, the binarized image determination submodule specifically includes:

[0243] A binary image acquisition unit, used to acquire a preliminary binary image corresponding to the original image;

[0244] A binary image scaling unit is used to scale the preliminary binary image corresponding to the original image in equal proportion to obtain a scaled binary image, wherein the length of the scaled binary image reaches a preset threshold;

[0245] The binary image filling unit is used to fill a preset third pixel value in the scaled binary image to obtain the binary image, and the length and width of the binary image both reach the preset threshold.

[0246] In an optional implementation, the binary image acquisition unit is specifically used to:

[0247] The element area is filled with a preset first pixel value, and the non-element area in the original image is filled with a preset second pixel value, so as to obtain a binary image corresponding to the original image.

[0248] In an optional implementation, the ratio determination submodule specifically includes:

[0249] A coordinate and parameter splicing unit, used for splicing the vertex coordinates and the geometric parameters into a one-dimensional vector;

[0250] A ratio determination unit is used to determine the restored geometric ratio of the element area according to the one-dimensional vector and the binarized image.

[0251] In an optional implementation, the ratio determination unit is specifically configured to:

[0252] The one-dimensional vector and the binarized image are input into a pre-trained geometric proportion prediction network to obtain the restored geometric proportion of the element region.

[0253] In an optional embodiment, the pre-trained geometric proportion prediction network includes a pre-trained convolutional neural network, a pre-trained first multi-layer perceptron, and a pre-trained second multi-layer perceptron; the proportion determination unit is specifically used for:

[0254] Inputting the binarized image into the pre-trained convolutional neural network to obtain an image vector;

[0255] Inputting the one-dimensional vector into the pre-trained first multi-layer perceptron to obtain a geometric vector;

[0256] Splicing the image vector with the geometric vector to obtain a spliced ​​vector;

[0257] The splicing vector is input into the pre-trained second multi-layer perceptron to obtain the restored geometric proportion of the element area.

[0258] In an optional implementation, the geometric ratio restoration is to restore the length-width ratio; the region restoration module specifically includes:

[0259] A length-width determination submodule, used to determine a target length and a target width of the element region according to the restored length-width ratio;

[0260] A coordinate construction submodule, used for constructing the restored vertex coordinates of the element area according to the target length and the target width;

[0261] A region transformation submodule, used for performing proportional restoration on the element region according to the vertex coordinates and the restored vertex coordinates to obtain a target element region;

[0262] The area transformation submodule specifically includes:

[0263] A matrix determination unit, used for determining a perspective transformation matrix according to the vertex coordinates and the restored vertex coordinates;

[0264] The region transformation unit is used to perform perspective transformation on the element region according to the perspective transformation matrix to obtain a target element region.

[0265] In an optional implementation, the length and width determination submodule specifically includes:

[0266] an original aspect ratio determining unit, used to determine the original aspect ratio of the element region;

[0267] A ratio comparison unit, used to compare the restored aspect ratio with the original aspect ratio;

[0268] The length and width determination unit is used to determine the target length and target width of the element area according to the comparison result.

[0269] In an optional implementation, the original aspect ratio determination unit is specifically configured to:

[0270] Obtaining a maximum length and a maximum width of the element region, and determining a first ratio between the maximum length and the maximum width;

[0271] The first ratio is determined as the original aspect ratio of the element region.

[0272] In an optional implementation, the length and width determination unit is specifically configured to:

[0273] In a case where the original aspect ratio is greater than or equal to the restored aspect ratio, determining the maximum length of the element region as the target length of the element region; determining a second ratio between the target length and the restored aspect ratio, and determining the second ratio as the target width of the element region;

[0274] Alternatively, when the original aspect ratio is smaller than the restored aspect ratio, the maximum width of the element region is determined as the target width of the element region; the product between the target width and the restored aspect ratio is determined, and the product is determined as the target length of the element region.

[0275] In an optional implementation, the matrix determination unit is specifically configured to:

[0276] The vertex coordinates and the restored vertex coordinates are input into a perspective transformation formula, and the perspective transformation formula is solved to obtain a perspective transformation matrix.

[0277] In an optional implementation, the region transformation unit is specifically configured to:

[0278] For any pixel in the element area, inputting the pixel coordinates of the pixel into a perspective transformation formula to obtain the restored coordinates corresponding to the pixel;

[0279] The perspective transformation formula is:

[0280] [x, y, c] = [u, v, c]*M;

[0281] Wherein, x and y are pixel coordinates of the pixel, u and v are restored coordinates, M is a perspective transformation matrix, and c is a constant;

[0282] A target element region is formed according to the restored coordinates of any pixel in the element region.

[0283] In an optional embodiment, the device further includes: a network training module, specifically configured to:

[0284] Acquire a sample image and a true geometric ratio of a sample element in the sample image, and determine a sample element area of ​​the sample element in the sample image;

[0285] Acquire sample geometric parameters of the sample image and sample vertex coordinates of the sample element area;

[0286] Determine a sample binary image corresponding to the sample image, and concatenate the sample vertex coordinates and the sample geometric parameters into a sample one-dimensional vector;

[0287] Inputting the sample one-dimensional vector and the sample binary image into a geometric ratio prediction network to obtain a predicted geometric ratio of the sample element region;

[0288] Determining a proportional loss between the predicted geometric proportion and the true geometric proportion, and training the geometric proportion prediction network based on the proportional loss;

[0289] When the scale loss converges, the training is stopped to obtain a pre-trained geometric scale prediction network.

[0290] The present application also provides an electronic device, such as Fig. 9 As shown, it includes a processor 91, a communication interface 92, a memory 93 and a communication bus 94, wherein the processor 91, the communication interface 92, and the memory 93 communicate with each other through the communication bus 94.

[0291] Memory 93, used for storing computer programs;

[0292] The processor 91 is used to execute the program stored in the memory 93 to implement the following steps:

[0293] Acquire an original image and determine an element region of a target element in the original image; acquire geometric parameters of the original image and vertex coordinates of the element region; determine a restored geometric proportion of the element region based on the vertex coordinates and the geometric parameters; and perform proportional restoration of the element region based on the restored geometric proportion to obtain a target element region.

[0294] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0295] The communication interface is used for communication between the above electronic device and other devices.

[0296] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0297] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0298] In another embodiment provided in the present application, a storage medium is provided. The storage medium stores instructions. When the instructions are executed on a computer, the computer executes the image processing method described in any one of the above embodiments.

[0299] In another embodiment provided in the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute the image processing method described in any one of the above embodiments.

[0300] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a storage medium, or transmitted from one storage medium to another storage medium, for example, the computer instructions may be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk (SSD)), etc.

[0301] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0302] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0303] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire an original image, and determine an element region of a target element in the original image; Acquire geometric parameters of the original image and vertex coordinates of the element area; Determining the restored geometric proportion of the element region according to the vertex coordinates and the geometric parameters; Based on the restored geometric proportion, the element region is restored in proportion to obtain a target element region.

2. The method according to claim 1, characterized in that The step of determining the restored geometric proportion of the element region according to the vertex coordinates and the geometric parameters includes: A binary image corresponding to the original image is determined, and a restored geometric proportion of the element region is determined according to the binary image, the vertex coordinates and the geometric parameters.

3. The method according to claim 2, characterized in that The step of determining a binary image corresponding to the original image comprises: Obtaining a preliminary binary image corresponding to the original image, scaling the preliminary binary image corresponding to the original image in equal proportion to obtain a scaled binary image, wherein the length of the scaled binary image reaches a preset threshold; A preset third pixel value is filled in the scaled binary image to obtain the binary image, wherein the length and width of the binary image both reach the preset threshold.

4. The method according to claim 3, characterized in that The step of obtaining a preliminary binary image corresponding to the original image includes: The element area is filled with a preset first pixel value, and the non-element area in the original image is filled with a preset second pixel value, so as to obtain a preliminary binarized image corresponding to the original image.

5. The method according to claim 2, characterized in that: The step of determining the restored geometric proportion of the element region according to the binary image, the vertex coordinates and the geometric parameters includes: The vertex coordinates and the geometric parameters are spliced ​​into a one-dimensional vector, and the restored geometric proportion of the element region is determined according to the one-dimensional vector and the binarized image.

6. The method according to claim 5, characterized in that The step of determining the restored geometric proportion of the element region according to the one-dimensional vector and the binarized image includes: The one-dimensional vector and the binarized image are input into a pre-trained geometric proportion prediction network to obtain the restored geometric proportion of the element region.

7. The method according to claim 6, characterized in that The pre-trained geometric proportion prediction network includes a pre-trained convolutional neural network, a pre-trained first multi-layer perceptron, and a pre-trained second multi-layer perceptron; The step of inputting the one-dimensional vector and the binarized image into a pre-trained geometric proportion prediction network to obtain the restored geometric proportion of the element region includes: Inputting the binarized image into the pre-trained convolutional neural network to obtain an image vector; Inputting the one-dimensional vector into the pre-trained first multi-layer perceptron to obtain a geometric vector; Splicing the image vector with the geometric vector to obtain a spliced ​​vector; The splicing vector is input into the pre-trained second multi-layer perceptron to obtain the restored geometric proportion of the element area.

8. The method according to claim 1, characterized in that The restored geometric proportion is to restore the length-width ratio; The method of performing proportional restoration on the element region based on the restored geometric proportion to obtain a target element region includes: Determining a target length and a target width of the element region according to the restored aspect ratio; Constructing restored vertex coordinates of the element area according to the target length and the target width; According to the vertex coordinates and the restored vertex coordinates, the element region is proportionally restored to obtain a target element region; The method of performing proportional restoration on the element region according to the vertex coordinates and the restored vertex coordinates to obtain a target element region includes: Determine a perspective transformation matrix according to the vertex coordinates and the restored vertex coordinates; According to the perspective transformation matrix, the element region is perspective transformed to obtain a target element region.

9. The method according to claim 8, characterized in that Determining the target length and target width of the element region according to the restored aspect ratio includes: Determining the original aspect ratio of the element region, and comparing the restored aspect ratio with the original aspect ratio; According to the comparison result, the target length and target width of the element area are determined.

10. The method according to claim 9, characterized in that The determining the original aspect ratio of the element region includes: Obtaining a maximum length and a maximum width of the element region, and determining a first ratio between the maximum length and the maximum width; The first ratio is determined as the original aspect ratio of the element region.

11. The method according to claim 9, characterized in that Determining the target length and target width of the element area according to the comparison result includes: In a case where the original aspect ratio is greater than or equal to the restored aspect ratio, determining the maximum length of the element region as the target length of the element region; determining a second ratio between the target length and the restored aspect ratio, and determining the second ratio as the target width of the element region; Alternatively, when the original aspect ratio is smaller than the restored aspect ratio, the maximum width of the element region is determined as the target width of the element region; the product between the target width and the restored aspect ratio is determined, and the product is determined as the target length of the element region.

12. The method according to claim 8, characterized in that The determining of the perspective transformation matrix according to the vertex coordinates and the restored vertex coordinates includes: The vertex coordinates and the restored vertex coordinates are input into a perspective transformation formula, and the perspective transformation formula is solved to obtain a perspective transformation matrix.

13. The method according to claim 8, characterized in that The step of performing perspective transformation on the element region according to the perspective transformation matrix to obtain a target element region includes: For any pixel in the element area, inputting the pixel coordinates of the pixel into a perspective transformation formula to obtain the restored coordinates corresponding to the pixel; The perspective transformation formula is: [x, y, c] = [u, v, c]*M; Wherein, x and y are pixel coordinates of the pixel, u and v are restored coordinates, M is a perspective transformation matrix, and c is a constant; A target element region is formed according to the restored coordinates of any pixel in the element region.

14. The method according to claim 6, characterized in that The pre-trained geometric ratio prediction network is obtained by: Acquire a sample image and a true geometric ratio of a sample element in the sample image, and determine a sample element area of ​​the sample element in the sample image; Acquire sample geometric parameters of the sample image and sample vertex coordinates of the sample element area; Determine a sample binary image corresponding to the sample image, and concatenate the sample vertex coordinates and the sample geometric parameters into a sample one-dimensional vector; Inputting the sample one-dimensional vector and the sample binary image into a geometric ratio prediction network to obtain a predicted geometric ratio of the sample element region; Determining a proportional loss between the predicted geometric proportion and the true geometric proportion, and training the geometric proportion prediction network based on the proportional loss; When the scale loss converges, the training is stopped to obtain a pre-trained geometric scale prediction network.

15. An image processing device, characterized in that: The device comprises: An image acquisition module, used to acquire an original image and determine an element region of a target element in the original image; A parameter and coordinate acquisition module, used for acquiring the geometric parameters of the original image and the vertex coordinates of the element area; A proportion determination module, used for determining the restored geometric proportion of the element area according to the vertex coordinates and the geometric parameters; The region restoration module is used to restore the element region in proportion based on the restored geometric proportion to obtain a target element region.

16. A storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 14 is implemented.

Images