Image tilt correction method, device and storage medium

Abstract

This application discloses a method, device, and storage medium for image tilt correction. The method comprises: obtaining an image to be corrected containing a graphic region of a target object, wherein the target object is rectangular; utilizing a deep learning image positioning model to perform image positioning on the image to be corrected, generating a mask image for segmenting the graphic region from the image to be corrected; determining first position information of the graphic region in the image to be corrected based on the mask image; and performing tilt correction on the graphic region in the image to be corrected based on the first position information.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image tilt correction method, device and storage medium. Background Art

[0002] The development of mobile internet has brought people closer together, and closer together with information. For OCR recognition, the key lies in accurately locating the text area to be recognized. However, when users upload images containing target objects (such as ID cards, bank cards, driving licenses, social security cards, bills, invoices, etc.) via mobile devices, the image area containing the target object often appears tilted or perspective due to the camera's shooting angle. This significantly affects text positioning and OCR recognition accuracy. Therefore, before performing OCR recognition, the image area containing the target object needs to be tilted and corrected.

[0003] Currently, the commonly used tilt correction method is based on traditional image processing. First, the image containing the target object is grayscaled and binarized to obtain an image in which the target object area is white and the background area is black. Next, edge detection is performed to obtain the four edges of the target object area. Then, edge straight line fitting is performed to obtain the coordinates of the four vertices of the target object area. Finally, a perspective transformation is performed to obtain the positive target object area. However, this tilt correction method has the following disadvantages: it is easily affected by the image background and it is difficult to determine the segmentation threshold during binarization. Especially when the pixel values ​​of the background and the pixel values ​​of the target object area are close, the threshold is difficult to determine and it is often impossible to effectively separate the background area from the target object area. Therefore, no straight line can be fitted and perspective transformation cannot be performed. Therefore, this method lacks robustness.

[0004] The traditional image tilt correction method in the above-mentioned prior art is easily affected by the image background and cannot effectively divide the background area and the graphic area containing the target object. Therefore, there is a technical problem of insufficient robustness, and no effective solution has been proposed so far. Summary of the Invention

[0005] The embodiments of the present disclosure provide an image tilt correction method, device, and storage medium to at least solve the technical problem in the prior art that traditional image tilt correction methods are easily affected by the image background, cannot effectively separate the background area and the graphic area containing the target object, and therefore lack robustness.

[0006] According to one aspect of an embodiment of the present disclosure, a method for image tilt correction is provided, including: obtaining an image to be corrected containing a graphic area of ​​a target object, wherein the target object is rectangular in shape; performing image positioning on the image to be corrected using a deep learning image positioning model, and generating a mask image for segmenting the graphic area from the image to be corrected; determining first position information of the graphic area in the image to be corrected based on the mask image; and performing tilt correction processing on the graphic area in the image to be corrected based on the first position information.

[0007] According to another aspect of an embodiment of the present disclosure, a storage medium is further provided, the storage medium including a stored program, wherein when the program is run, a processor executes any one of the above methods.

[0008] According to another aspect of an embodiment of the present disclosure, an image tilt correction device is also provided, including: an acquisition module for acquiring an image to be corrected containing a graphic area of ​​a target object, wherein the target object is rectangular in shape; an image positioning module for performing image positioning on the image to be corrected using a deep learning image positioning model, and generating a mask image for segmenting the graphic area from the image to be corrected; a first position information determination module for determining the first position information of the graphic area in the image to be corrected based on the mask image; and a tilt correction module for performing tilt correction processing on the graphic area in the image to be corrected based on the first position information.

[0009] According to another aspect of an embodiment of the present disclosure, an image tilt correction device is also provided, including: a processor; and a memory connected to the processor, for providing the processor with instructions for processing the following processing steps: obtaining an image to be corrected containing a graphic area of ​​a target object, wherein the target object is rectangular in shape; performing image positioning on the image to be corrected using a deep learning image positioning model, and generating a mask image for segmenting the graphic area from the image to be corrected; determining first position information of the graphic area in the image to be corrected based on the mask image; and performing tilt correction processing on the graphic area in the image to be corrected based on the first position information.

[0010] In the disclosed embodiment, an image to be corrected containing a graphic region of a target object is first acquired. Then, a deep learning image positioning model is used to perform image positioning on the image to be corrected, generating a mask image (mask image) for segmenting the graphic region from the image to be corrected. Second, based on the mask image, first position information of the graphic region in the image to be corrected is determined. Finally, based on the first position information, the graphic region in the image to be corrected is subjected to tilt correction processing. In this embodiment, the position information of the graphic region containing the target object in the image to be corrected is no longer determined by grayscale conversion and binary threshold segmentation. Instead, a deep learning image positioning model is used to generate a mask image for segmenting the graphic region from the image to be corrected. Then, based on the mask image, the position information of the graphic region in the image to be corrected is determined. Therefore, compared with existing image tilt correction methods, the present invention is not easily affected by the background, can effectively separate the background area and the graphic region containing the target object, has good robustness and high accuracy, and can effectively overcome the technical effect of complex background conditions. This solves the technical problem that the traditional image tilt correction method in the prior art is easily affected by the image background and cannot effectively divide the background area and the graphic area containing the target object, thus lacking robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The drawings described herein are used to provide a further understanding of the present disclosure and constitute a part of this application. The illustrative embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation of the present disclosure. In the drawings:

[0012] Figure 1 is a hardware structure block diagram of a computing device for implementing the method according to embodiment 1 of the present disclosure;

[0013] Figure 2 is a flowchart of the image tilt correction method according to the first aspect of Embodiment 1 of the present disclosure;

[0014] Figure 3 1 is a schematic diagram of the overall process of the image tilt correction method according to the first aspect of Embodiment 1 of the present disclosure;

[0015] Figure 4 is a schematic diagram of the network structure of the image positioning model according to the first aspect of embodiment 1 of the present disclosure;

[0016] Figure 5 is a structural diagram of the bneck unit according to the first aspect of embodiment 1 of the present disclosure;

[0017] Figure 6 is a schematic diagram of the image tilt correction device according to embodiment 2 of the present disclosure; and

[0018] Figure 7 2 is a schematic diagram of an image tilt correction device according to embodiment 3 of the present disclosure. DETAILED DESCRIPTION

[0019] In order to enable those skilled in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present disclosure.

[0020] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0021] First, some nouns or terms that appear in the description of the embodiments of the present disclosure are subject to the following explanations:

[0022] OCR: (Optical Character Recognition), commonly used to recognize characters in images.

[0023] BN: short for Batch Normalization, often used after the convolutional layer of a neural network to force the increasingly biased distribution back to a more standard distribution, which can speed up training and improve the generalization ability of the model.

[0024] relu: An activation function commonly used in deep learning, which has the effect of speeding up training and preventing gradient disappearance.

[0025] Example 1

[0026] According to this embodiment, an embodiment of an image tilt correction method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0027] The method embodiment provided in this embodiment can be executed in a mobile terminal, a computer terminal, a server or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computing device for implementing an image tilt correction method. Figure 1 As shown, the computing device may include one or more processors (the processor may include but is not limited to a microprocessor MCU or a programmable logic device FPGA, etc.), a memory for storing data, and a transmission device for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a power supply and / or a camera. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0028] It should be noted that the one or more processors and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry." The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuitry may be a single, independent processing module, or may be incorporated in whole or in part into any of the other components of the computing device. As described in the embodiments of the present disclosure, the data processing circuitry serves as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0029] The memory can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the image tilt correction method in the embodiment of the present disclosure. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, implementing the image tilt correction method of the above-mentioned application. The memory may include a high-speed random access memory and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely located relative to the processor, and these remote memories may be connected to the computing device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0030] The transmission device is used to receive or send data via a network. Specific examples of the aforementioned network may include a wireless network provided by a communications provider of the computing device. In one embodiment, the transmission device includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission device may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0031] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computing device.

[0032] It should be noted that, in some optional embodiments, the above Figure 1 The computing device shown may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of both hardware and software elements. Figure 1 This is merely one example of a particular embodiment and is intended to illustrate the types of components that may be present in the computing devices described above.

[0033] In the above operating environment, according to the first aspect of this embodiment, a method for image tilt correction is provided. Figure 2 A schematic diagram showing the process of the method is shown in FIG. Figure 2 As shown, the method includes:

[0034] S202: Acquire an image to be corrected in a graphic area containing a target object, wherein the target object is in a rectangular shape;

[0035] S204: Using a deep learning image positioning model, perform image positioning on the image to be corrected, and generate a mask image for segmenting a graphic area from the image to be corrected;

[0036] S206: Determine first position information of the graphic region in the image to be corrected according to the mask image; and

[0037] S208: Performing tilt correction processing on the graphic area in the image to be corrected according to the first position information.

[0038] As described in the previous background technology, the currently commonly used tilt correction method is based on traditional image processing methods. First, the image containing the target object is grayscaled and binarized to obtain an image in which the graphic area containing the target object is white and the background area is black. Next, edge detection is performed to obtain the four edges of the graphic area containing the target object. Then, edge straight line fitting is performed to obtain the four vertex coordinates of the graphic area containing the target object. Finally, a perspective transformation is performed to obtain the positive target object area. However, this tilt correction method has the following disadvantages: it is easily affected by the image background, and it is difficult to determine the segmentation threshold when binarization is performed. In particular, when the pixel values ​​of the background and the pixel values ​​of the graphic area containing the target object are relatively close, the threshold is not easy to determine, and it is often impossible to effectively separate the background area and the graphic area containing the target object. Therefore, no straight line can be fitted and no perspective transformation can be performed. Therefore, this method lacks robustness.

[0039] In view of the problems existing in the above background technology, combined with Figure 2 As shown, this embodiment provides an image tilt correction method that can be deployed on a mobile terminal, has a fast correction speed, and is highly robust. Figure 2 As shown, in this embodiment, firstly, an image to be corrected containing a graphic area of ​​a target object is obtained, wherein the target object is a rectangular shape. The target object is, for example but not limited to: an ID card, a bank card, a driving license, a social security card, a bill, an invoice, etc. For example, refer to Figure 3 As shown, the target object of this embodiment is an ID card, and the image to be corrected is an ID card image. Due to the problem of shooting angle, the ID card area in the image has a certain tilt, so the ID card area in the ID card image needs to be tilt corrected.

[0040] Furthermore, the image positioning model of deep learning is used to perform image positioning on the image to be corrected, and a mask image for segmenting the graphic area from the image to be corrected is generated. The image positioning model of deep learning can be, for example, a positioning network that integrates the Mobilenet concept of a lightweight network with deep learning and a simple structure, and the network is obtained through a large number of image training. Figure 3As shown in the figure, after the image to be corrected is input into the image localization model, the ID card area in the mask image output by the image localization model is white (with non-zero pixels) and the background area is black (with zero pixels). In other words, the ID card area (corresponding to the aforementioned graphic area containing the target object) and the background area in the mask image output by the image localization model are clearly separated.

[0041] Furthermore, based on the mask image, first position information of the graphic region in the image to be corrected is determined. Specifically, because the graphic region and background region in the mask image are clearly demarcated, the position information of the graphic region in the mask image can be well determined. Consequently, based on the pre-set mapping relationship between the image to be corrected and the mask image, the first position information of the graphic region in the image to be corrected can be quickly and accurately determined. Finally, based on the first position information, tilt correction processing is performed on the graphic region in the image to be corrected.

[0042] Thus, in this manner, this embodiment first obtains an image to be corrected containing a graphic region of a target object. It then utilizes a deep learning image localization model to perform image localization on the image to be corrected, generating a mask image for segmenting the graphic region from the image to be corrected. Secondly, based on the mask image, it determines first position information of the graphic region in the image to be corrected. Finally, based on the first position information, it performs tilt correction on the graphic region in the image to be corrected. In this embodiment, the position information of the graphic region containing the target object in the image to be corrected is no longer determined through grayscale conversion and binary threshold segmentation. Instead, a deep learning image localization model is utilized to generate a mask image for segmenting the graphic region from the image to be corrected. Finally, based on the mask image, the position information of the graphic region in the image to be corrected is determined. Therefore, compared to existing image tilt correction methods, this method is less susceptible to background influences, can effectively separate the background region from the graphic region containing the target object, has good robustness, high accuracy, and can effectively overcome complex background conditions. This solves the technical problem that the traditional image tilt correction method in the prior art is easily affected by the image background and cannot effectively divide the background area and the graphic area containing the target object, thus lacking robustness.

[0043] Optionally, the operation of determining the first position information of the graphic area in the image to be corrected based on the mask image includes: determining the second position information of the non-zero pixel area in the mask image; and determining the first position information of the graphic area in the image to be corrected based on the second position information.

[0044] Specifically, in the process of determining the first position information of the graphic area in the image to be corrected based on the mask image, the second position information of the non-zero pixel area in the mask image is first determined. Figure 3As shown, the non-zero pixel area (i.e., white area) and the zero pixel area (i.e., black area) in the mask image are clearly divided. The non-zero pixel area is the graphic area containing the target object (corresponding to the ID card area), and the zero pixel area is the background area. Since the non-zero pixel area (i.e., white area) and the background area in the mask image are clearly divided, the second position information of the non-zero pixel area in the mask image can be quickly determined. Then, based on the second position information, the first position information of the graphic area in the image to be corrected is determined. In this way, the first position information of the graphic area in the image to be corrected can be determined quickly, accurately and effectively.

[0045] Optionally, the operation of determining the second position information of the non-zero pixel area in the mask image includes: performing edge detection on the mask image to obtain a set of points on the four edges of the non-zero pixel area; and performing straight line fitting based on the set of points on the four edges to determine the coordinate information of four corner points corresponding to the four corner points of the non-zero pixel area as the second position information.

[0046] Specifically, refer to Figure 3 As shown, in the process of determining the second position information of the non-zero pixel region in the mask image, edge detection techniques commonly used in image processing are first used to perform edge detection on the mask image, obtaining a set of points on the four edges of the non-zero pixel region. Linear fitting is then performed on the points on the four edges to obtain four straight lines. These four straight lines represent the four sides of the non-zero pixel region. The intersection points of each pair of straight lines are then calculated to obtain the coordinate information of the four corner points corresponding to the four corner points of the non-zero pixel region. Finally, the obtained coordinate information of the four corner points is used as the second position information. In this way, the second position information of the non-zero pixel region in the mask image can be accurately determined.

[0047] Optionally, the operation of performing tilt correction processing on the graphic area in the image to be corrected according to the first position information includes: performing perspective transformation on the graphic area in the image to be corrected according to the first position information.

[0048] Specifically, refer to Figure 3 As shown, based on the first position information, the graphic area in the corrected image (i.e., the ID card image) is perspective transformed to obtain a positive ID card image. In this way, the graphic area in the corrected image is effectively tilt-corrected to facilitate subsequent text positioning and OCR recognition operations.

[0049] Optionally, using a deep learning image positioning model to perform image positioning on the image to be corrected and generate a mask image for segmenting a graphic area from the image to be corrected, the operation includes: generating an input image corresponding to the image to be corrected according to a preset scaling ratio, wherein the input image is suitable for image positioning by the image positioning model, and the size of the input image is equal to the size of the mask image; and using the image positioning model to perform image positioning on the input image to generate a mask image.

[0050] Specifically, the size of the image to be corrected is typically 1080*1244. However, because image localization models often have input image size requirements (for example, requiring the input image to be 256*256), the image to be corrected must be scaled according to a preset scaling ratio before being input into the image localization model. This generates an input image suitable for image localization by the image localization model. The image localization model is then used to perform image localization on the input image to generate a mask image. This ensures the accuracy of the generated mask image.

[0051] Optionally, the operation of determining the first position information of the graphic area in the image to be corrected according to the second position information includes: determining the first position information of the graphic area in the image to be corrected according to the second position information and a preset zoom ratio.

[0052] Specifically, referring to the above, in order to generate an input image suitable for image positioning by the image positioning model, this embodiment scales the image to be corrected according to a preset scaling ratio before inputting it into the image positioning model. Therefore, in determining the first position information of the graphic region in the image to be corrected based on the second position information, it is necessary to perform a coordinate transformation on the second position information based on the second position information and the preset scaling ratio. Specifically, the first position information of the graphic region in the image to be corrected is determined based on the mapping relationship between the image to be corrected and the mask image. This ensures the accuracy of the determined first position information.

[0053] Optionally, the method further includes training the image positioning model through the following operations: obtaining a plurality of sample images of a graphic area to be corrected containing a rectangular shape; constructing an image positioning model, wherein the image positioning model is used to generate an output vector, and the output vector is used to indicate the coordinate information of the graphic area in the sample image; using the image positioning model, generating an output vector corresponding to the sample image; and comparing the output vector with a pre-set annotation vector corresponding to the sample image, and adjusting the image positioning model according to the comparison result, wherein the annotation vector is used to indicate the actual coordinate information of the graphic area in the sample image.

[0054] Specifically, during the data preprocessing phase (i.e., acquiring sample images), 3,000 sample images containing rectangular graphic regions (such as, but not limited to, user ID card images) were collected and organized. These images were annotated, primarily with the coordinates of the four corner points of the ID card region. In this implementation, data augmentation (including rotation and lighting changes) was performed on the original 3,000 annotated images, increasing the total to 30,000 images, of which 5,000 were reserved for the test set. The original images and labeled images were combined to form the training set.

[0055] Furthermore, an image positioning model is constructed. The image positioning model is used to generate an output vector, which indicates the coordinate information of the graphic region in the sample image. The image positioning model is then used to generate an output vector corresponding to the sample image. Finally, the output vector is compared with a pre-set annotation vector corresponding to the sample image, and the image positioning model is adjusted based on the comparison result. The annotation vector indicates the actual coordinate information of the graphic region in the sample image. In this way, the image positioning model is continuously adjusted and optimized to achieve the best positioning effect.

[0056] Optionally, the operation of comparing the output vector with a pre-set annotation vector corresponding to the sample image includes calculating the L1 spatial distance between the output vector and the annotation vector, and adjusting the image positioning model according to the comparison result, including: using the L1 spatial distance as a loss function, calculating the gradient of the loss function; and adjusting the image positioning model based on the gradient according to the stochastic gradient descent principle. For example: using the PQ method, asymmetric distance computation (ADC) based on a lookup table is used to quickly calculate the distance between target feature vectors. Under the same compression ratio, the retrieval accuracy of the PQ method using ADC is higher than that of the binary encoding method using Hamming distance. Then, using the L1 spatial distance as a loss function, the gradient of the loss function is calculated and the network parameters are optimized according to the stochastic gradient descent principle (SGD).

[0057] In addition, it should be noted that the model structure of the present invention is simple and not complicated, the final model size is only 350KB, and the correction speed is at the millisecond level, so it can be used on mobile terminals. Figure 4 The network structure diagram of the image positioning model is shown as an example. Figure 4As shown in the figure, after scaling the image to be rectified, the input image of 256*256*3 is obtained. When the input image of 256*256*3 is input into the image localization model, it is first converted into a 128*128*64 image by a depthwise separable convolution with a 3*3 convolution kernel, and then passes through 3 bneck units to obtain a 16*16*512 feature map. This process can be called the encoding process, refer to Figure 4 As shown in FIG, in the encoding process, in order to prevent the loss of image information during the convolution process, a direct connection operation (shortcut operation) in the residual network model can be used to perform a direct connection operation on each network layer.

[0058] Furthermore, the 16*16*512 feature map obtained after encoding is decoded through four layers of 3*3 convolution kernel deconvolution, ultimately obtaining a 256*256*1 feature map. The deconvolution method is also depthwise separable convolution. In addition, during the encoding and decoding process, to prevent image information loss due to increased convolution, the current feature map is fused with the feature map of the previous network layer through skip connections.

[0059] Further, Figure 5 An example is shown Figure 4 The structure of the bneck unit in . Figure 4 as well as Figure 5 As shown in the figure, in each bneck unit, the image first undergoes 1*1 convolution, BN and activation function relu, then passes 3*3 depth-separable convolution, BN and relu, and finally undergoes 1*1 convolution and BN.

[0060] In addition, the image tilt correction method proposed in this embodiment adds an image positioning model on the basis of existing traditional image processing, and the image positioning model uses deep learning to perform end-to-end learning prediction. In addition, this embodiment makes three improvements to the image positioning model: 1) Drawing on the structure of the encoding and decoding of the unet network, a network structure with 4 layers of encoding and 4 layers of decoding is designed; 2) Drawing on the structure of the Mobilenet network, the convolution process in the encoding process is replaced by the bneck unit, but the SE module is removed; 3) All convolutions in the decoding process are replaced by depthwise separable convolutions. These three improvements speed up the training speed and reduce the size of the image positioning model.

[0061] In summary, the image tilt correction method proposed in this embodiment can produce the following beneficial effects:

[0062] 1. Deployable on mobile devices: The image localization model used in this embodiment has a simple network structure and uses many 1*1 convolutions, which reduces the number of parameters. In addition, all ordinary convolutions are replaced with depthwise separable convolutions. This replacement reduces the network parameters by about 1 / 9, so the final model size is only 350KB, which meets the size requirements for mobile deployment.

[0063] 2. It can help improve the accuracy of ID card OCR recognition: For OCR recognition, if tilt correction is not performed, the located text area is often tilted due to the problem of shooting angle, and the result with perspective transformation often affects the accuracy of OCR recognition.

[0064] Fast correction speed and strong robustness: Based on a deep learning process, the image localization model used in this implementation is an end-to-end network model that is not easily affected by background noise. It has good robustness and high accuracy, and can effectively overcome complex background conditions. In addition, the model is simple and the correction speed is fast, in milliseconds.

[0065] In addition, reference Figure 1 As shown, according to the second aspect of this embodiment, a storage medium is provided, which includes a stored program, wherein when the program is run, a processor executes any one of the above methods.

[0066] Therefore, according to this embodiment, an image to be corrected containing a graphic region of a target object is first acquired. A deep learning image localization model is then used to perform image localization on the image to be corrected, generating a mask image (mask image) for segmenting the graphic region from the image to be corrected. Second, based on the mask image, first position information of the graphic region in the image to be corrected is determined. Finally, based on the first position information, the graphic region in the image to be corrected is subjected to tilt correction. In this embodiment, the position information of the graphic region containing the target object in the image to be corrected is no longer determined by grayscale conversion and binary threshold segmentation. Instead, a deep learning image localization model is used to generate a mask image for segmenting the graphic region from the image to be corrected. Based on the mask image, the position information of the graphic region in the image to be corrected is determined. Therefore, compared to existing image tilt correction methods, this method is less susceptible to background influences, can effectively separate the background region from the graphic region containing the target object, has good robustness, high accuracy, and can effectively overcome complex background conditions. This solves the technical problem that the traditional image tilt correction method in the prior art is easily affected by the image background and cannot effectively divide the background area and the graphic area containing the target object, thus lacking robustness.

[0067] It should be noted that for the aforementioned method embodiments, for simplicity of description, they are all expressed as a series of action combinations. However, those skilled in the art should be aware that the present invention is not limited by the order of the actions described, because according to the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0068] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0069] Example 2

[0070] Figure 6 FIG. 6 shows an image tilt correction device 600 according to this embodiment, which corresponds to the method according to the first aspect of embodiment 1. Figure 6 As shown, the device 600 includes: an acquisition module 610, used to acquire an image to be corrected containing a graphic area of ​​a target object, wherein the target object is rectangular in shape; an image positioning module 620, used to perform image positioning on the image to be corrected using an image positioning model of deep learning, and generate a mask image for segmenting the graphic area from the image to be corrected; a first position information determination module 630, used to determine the first position information of the graphic area in the image to be corrected based on the mask image; and a tilt correction module 640, used to perform tilt correction processing on the graphic area in the image to be corrected based on the first position information.

[0071] Optionally, the first position information determination module 630 includes: a first determination submodule for determining the second position information of the non-zero pixel area in the mask image; and a second determination submodule for determining the first position information of the graphic area in the image to be corrected based on the second position information.

[0072] Optionally, the first determination submodule includes: a first determination unit, used to perform edge detection on the mask image to obtain a set of points on the four edges of the non-zero pixel area; and a second determination unit, used to perform straight line fitting based on the set of points on the four edges to determine the coordinate information of four corner points corresponding to the four corner points of the non-zero pixel area as the second position information.

[0073] Optionally, the tilt correction module 640 includes: a tilt correction submodule, configured to perform perspective transformation on the graphic region in the image to be corrected according to the first position information.

[0074] Optionally, the image positioning module 620 includes: a first generation submodule, used to generate an input image corresponding to the image to be corrected according to a preset scaling ratio, wherein the input image is suitable for image positioning by the image positioning model, and the size of the input image is equal to the size of the mask image; and a second generation submodule, used to use the image positioning model to perform image positioning on the input image to generate a mask image.

[0075] Optionally, the second determining submodule includes: a third determining unit, configured to determine first position information of the graphic region in the image to be corrected according to the second position information and a preset zoom ratio.

[0076] Optionally, a training module is also included for training the image positioning model through the following operations: obtaining multiple sample images of the graphic area to be corrected containing a rectangular shape; constructing an image positioning model, wherein the image positioning model is used to generate an output vector, and the output vector is used to indicate the coordinate information of the graphic area in the sample image; using the image positioning model, generating an output vector corresponding to the sample image; and comparing the output vector with a pre-set annotation vector corresponding to the sample image, and adjusting the image positioning model according to the comparison result, wherein the annotation vector is used to indicate the actual coordinate information of the graphic area in the sample image.

[0077] Therefore, according to this embodiment, an image to be corrected containing a graphic region of a target object is first acquired. A deep learning image localization model is then used to perform image localization on the image to be corrected, generating a mask image for segmenting the graphic region from the image to be corrected. Second, based on the mask image, first position information of the graphic region in the image to be corrected is determined. Finally, based on the first position information, the graphic region in the image to be corrected is subjected to tilt correction. In this embodiment, the position information of the graphic region containing the target object in the image to be corrected is no longer determined through grayscale conversion and binary threshold segmentation. Instead, a deep learning image localization model is used to generate a mask image for segmenting the graphic region from the image to be corrected. Based on the mask image, the position information of the graphic region in the image to be corrected is determined. Therefore, compared to existing image tilt correction methods, this method is less susceptible to background influences, can effectively separate the background region from the graphic region containing the target object, has good robustness, high accuracy, and can effectively overcome complex background conditions. This solves the technical problem that the traditional image tilt correction method in the prior art is easily affected by the image background and cannot effectively divide the background area and the graphic area containing the target object, thus lacking robustness.

[0078] Example 3

[0079] Figure 7 FIG. 7 shows an image tilt correction device 700 according to this embodiment, which corresponds to the method according to the first aspect of embodiment 1. Figure 7 As shown, the device 700 includes: a processor 710; and a memory 720, which is connected to the processor 710 and is used to provide the processor 710 with instructions for processing the following processing steps: obtaining an image to be corrected of a graphic area containing a target object, wherein the target object is rectangular in shape; using a deep learning image positioning model to perform image positioning on the image to be corrected, and generate a mask image for segmenting the graphic area from the image to be corrected; determining first position information of the graphic area in the image to be corrected based on the mask image; and performing tilt correction processing on the graphic area in the image to be corrected based on the first position information.

[0080] Optionally, the operation of determining the first position information of the graphic area in the image to be corrected based on the mask image includes: determining the second position information of the non-zero pixel area in the mask image; and determining the first position information of the graphic area in the image to be corrected based on the second position information.

[0081] Optionally, the operation of determining the second position information of the non-zero pixel area in the mask image includes: performing edge detection on the mask image to obtain a set of points on the four edges of the non-zero pixel area; and performing straight line fitting based on the set of points on the four edges to determine the coordinate information of four corner points corresponding to the four corner points of the non-zero pixel area as the second position information.

[0082] Optionally, the operation of performing tilt correction processing on the graphic area in the image to be corrected according to the first position information includes: performing perspective transformation on the graphic area in the image to be corrected according to the first position information.

[0083] Optionally, using a deep learning image positioning model to perform image positioning on the image to be corrected and generate a mask image for segmenting a graphic area from the image to be corrected, the operation includes: generating an input image corresponding to the image to be corrected according to a preset scaling ratio, wherein the input image is suitable for image positioning by the image positioning model, and the size of the input image is equal to the size of the mask image; and using the image positioning model to perform image positioning on the input image to generate a mask image.

[0084] Optionally, the operation of determining the first position information of the graphic area in the image to be corrected according to the second position information includes: determining the first position information of the graphic area in the image to be corrected according to the second position information and a preset zoom ratio.

[0085] Optionally, the memory 720 is also used to provide instructions for the processor 710 to process the following processing steps: training the image positioning model through the following operations: obtaining multiple sample images of the graphic area to be corrected containing a rectangular shape; constructing an image positioning model, wherein the image positioning model is used to generate an output vector, and the output vector is used to indicate the coordinate information of the graphic area in the sample image; using the image positioning model, generating an output vector corresponding to the sample image; and comparing the output vector with a pre-set annotation vector corresponding to the sample image, and adjusting the image positioning model according to the comparison result, wherein the annotation vector is used to indicate the actual coordinate information of the graphic area in the sample image.

[0086] Therefore, according to this embodiment, an image to be corrected containing a graphic region of a target object is first acquired. A deep learning image localization model is then used to perform image localization on the image to be corrected, generating a mask image for segmenting the graphic region from the image to be corrected. Second, based on the mask image, first position information of the graphic region in the image to be corrected is determined. Finally, based on the first position information, the graphic region in the image to be corrected is subjected to tilt correction. In this embodiment, the position information of the graphic region containing the target object in the image to be corrected is no longer determined through grayscale conversion and binary threshold segmentation. Instead, a deep learning image localization model is used to generate a mask image for segmenting the graphic region from the image to be corrected. Based on the mask image, the position information of the graphic region in the image to be corrected is determined. Therefore, compared to existing image tilt correction methods, this method is less susceptible to background influences, can effectively separate the background region from the graphic region containing the target object, has good robustness, high accuracy, and can effectively overcome complex background conditions. This solves the technical problem that the traditional image tilt correction method in the prior art is easily affected by the image background and cannot effectively divide the background area and the graphic area containing the target object, thus lacking robustness.

[0087] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0088] In the above embodiments of the present invention, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0089] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0090] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0091] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0092] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk, etc. Various media that can store program codes.

[0093] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A method for image tilt correction, characterized in that: include: Acquire an image to be corrected of a graphic region containing a target object, wherein the target object is rectangular in shape; Performing image positioning on the image to be corrected using a deep learning image positioning model, and generating a mask image for segmenting the graphic area from the image to be corrected; determining first position information of the graphic region in the image to be corrected according to the mask image; as well as performing tilt correction processing on the graphic area in the image to be corrected according to the first position information; The operation of determining first position information of the graphic region in the image to be corrected according to the mask image includes: Determining second position information of a non-zero pixel area in the mask image; and determining the first position information of the graphic area in the image to be corrected according to the second position information; The operation of determining second position information of a non-zero pixel area in the mask image includes: Performing edge detection on the mask image to obtain a set of points on four edges of the non-zero pixel area; and Performing straight line fitting based on the set of points on the four edges to determine coordinate information of four corner points corresponding to the four corner points of the non-zero pixel area as the second position information; The operation of performing tilt correction processing on the graphic area in the image to be corrected according to the first position information includes: performing perspective transformation on the graphic area in the image to be corrected according to the first position information; The operation of performing image positioning on the image to be corrected by using a deep learning image positioning model and generating a mask image for segmenting the graphic area from the image to be corrected includes: generating an input image corresponding to the image to be corrected according to a preset scaling ratio, wherein the input image is suitable for image positioning by the image positioning model, and a size of the input image is equal to a size of the mask image; and Performing image positioning on the input image using the image positioning model to generate the mask image; The operation of determining the first position information of the graphic area in the image to be corrected according to the second position information includes: determining the first position information of the graphic area in the image to be corrected according to the second position information and the preset zoom ratio; The method further includes training the image positioning model by performing the following operations: Acquire multiple sample images of a graphic area to be corrected that includes a rectangular shape; constructing an image positioning model, wherein the image positioning model is used to generate an output vector, and the output vector is used to indicate coordinate information of a graphic area in the sample image; generating an output vector corresponding to the sample image using the image localization model; and The output vector is compared with a preset annotation vector corresponding to the sample image, and the image positioning model is adjusted according to the comparison result, wherein the annotation vector is used to indicate actual coordinate information of the graphic area in the sample image.

2. A storage medium, characterized in that: The storage medium includes a stored program, wherein when the program is executed, the method according to claim 1 is executed by a processor.

3. An image tilt correction device, characterized in that: include: an acquisition module, configured to acquire an image to be corrected of a graphic region containing a target object, wherein the target object is rectangular in shape; An image positioning module, configured to perform image positioning on the image to be corrected using a deep learning image positioning model, and generate a mask image for segmenting the graphic area from the image to be corrected; a first position information determining module, configured to determine first position information of the graphic region in the image to be corrected based on the mask image; as well as a tilt correction module, configured to perform tilt correction processing on the graphic area in the image to be corrected according to the first position information; The first location information determination module is specifically configured to: determining second position information of a non-zero pixel region in the mask image; and determining the first position information of the graphic region in the image to be corrected based on the second position information; Performing edge detection on the mask image to obtain a set of points on four edges of the non-zero pixel area; and performing straight line fitting according to the set of points on the four edges to determine coordinate information of four corner points corresponding to the four corner points of the non-zero pixel area as the second position information; The tilt correction module is specifically used for: performing perspective transformation on the graphic area in the image to be corrected according to the first position information; The image positioning module is specifically used for: generating an input image corresponding to the image to be corrected according to a preset scaling ratio, wherein the input image is suitable for image positioning by the image positioning model and a size of the input image is equal to a size of the mask image; and performing image positioning on the input image using the image positioning model to generate the mask image; The first location information determining module is further configured to: determining first position information of the graphic region in the image to be corrected according to the second position information and the preset scaling ratio; The acquisition module is further used to train the image positioning model through the following operations: Acquire multiple sample images of a graphic area to be corrected containing a rectangular shape; construct an image positioning model, wherein the image positioning model is used to generate an output vector, and the output vector is used to indicate the coordinate information of the graphic area in the sample image; use the image positioning model to generate an output vector corresponding to the sample image; and compare the output vector with a pre-set annotation vector corresponding to the sample image, and adjust the image positioning model according to the comparison result, wherein the annotation vector is used to indicate the actual coordinate information of the graphic area in the sample image.

4. An image tilt correction device, characterized in that: include: processor; as well as A memory, connected to the processor, configured to provide the processor with instructions for processing the following processing steps: Acquire an image to be corrected of a graphic region containing a target object, wherein the target object is rectangular in shape; Performing image positioning on the image to be corrected using a deep learning image positioning model, and generating a mask image for segmenting the graphic area from the image to be corrected; determining first position information of the graphic region in the image to be corrected according to the mask image; as well as performing tilt correction processing on the graphic area in the image to be corrected according to the first position information; The memory is further configured to provide the processor with instructions for performing the following processing steps: determining second position information of a non-zero pixel region in the mask image; and determining the first position information of the graphic region in the image to be corrected based on the second position information; Performing edge detection on the mask image to obtain a set of points on four edges of the non-zero pixel area; and performing straight line fitting according to the set of points on the four edges to determine coordinate information of four corner points corresponding to the four corner points of the non-zero pixel area as the second position information; performing perspective transformation on the graphic area in the image to be corrected according to the first position information; generating an input image corresponding to the image to be corrected according to a preset scaling ratio, wherein the input image is suitable for image positioning by the image positioning model and a size of the input image is equal to a size of the mask image; and performing image positioning on the input image using the image positioning model to generate the mask image; determining first position information of the graphic region in the image to be corrected according to the second position information and the preset scaling ratio; Acquire multiple sample images of a graphic area to be corrected that includes a rectangular shape; An image positioning model is constructed, wherein the image positioning model is used to generate an output vector, wherein the output vector is used to indicate coordinate information of a graphic area in the sample image; an output vector corresponding to the sample image is generated using the image positioning model; and the output vector is compared with a pre-set annotation vector corresponding to the sample image, and the image positioning model is adjusted according to the comparison result, wherein the annotation vector is used to indicate actual coordinate information of the graphic area in the sample image.

Images