Image correction method and device, electronic device and medium

Abstract

The present disclosure provides an image correction method and device, electronic equipment, computer readable storage medium and computer program product, relating to the technical field of artificial intelligence, especially to the technical field of computer vision, image processing, deep learning and the like, which can be applied to the scene of OCR, smart government affairs and the like. The implementation scheme is as follows: obtaining a to-be-corrected image; determining a deformation image feature of the to-be-corrected image, wherein the deformation image feature indicates a type and a degree of deformation of the to-be-corrected image; performing foreground-background segmentation on the to-be-corrected image to obtain background information of the to-be-corrected image; performing backward prediction on the to-be-corrected image according to the deformation image feature and the background information to obtain a backward image of the to-be-corrected image; and correcting the to-be-corrected image based on the backward image.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of artificial intelligence, in particular to the technical field of computer vision, image processing, deep learning and the like, which can be applied to the scene of OCR, smart government affairs, and specifically relates to an image correction method and device, electronic equipment, computer readable storage medium and computer program product. BACKGROUND

[0002] Artificial intelligence is a discipline that studies enabling computers to simulate some thinking processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.) of human beings, which has both hardware technology and software technology. Artificial intelligence hardware technology generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing, etc. Artificial intelligence software technology mainly includes computer vision technology, speech recognition technology, natural language processing technology, and machine learning / deep learning, big data processing technology, knowledge graph technology, etc. several major directions.

[0003] In the related art, the backward prediction method is usually used to correct the image that has deformation. However, for the deformed image with unclear or missing edge contour of the foreground subject, the backward prediction method has the problem of being unable to effectively determine the edge contour of the foreground subject in the deformed image, resulting in poor correction effect. SUMMARY

[0004] The present disclosure provides an image correction method, device, electronic equipment, computer readable storage medium and computer program product.

[0005] According to an aspect of the present disclosure, an image correction method is provided, comprising: obtaining a to-be-corrected image; determining a deformed image feature of the to-be-corrected image, wherein the deformed image feature indicates the type and degree of deformation of the to-be-corrected image; performing foreground and background segmentation on the to-be-corrected image to obtain background information of the to-be-corrected image; performing backward prediction on the to-be-corrected image according to the deformed image feature and the background information to obtain a backward image of the to-be-corrected image; and correcting the to-be-corrected image based on the backward image.

[0006] According to another aspect of the present disclosure, an image correction device is provided, comprising: an acquisition unit configured to acquire a to-be-corrected image; a determination unit configured to determine a deformation image feature of the to-be-corrected image, wherein the deformation image feature indicates a type and a degree of deformation of the to-be-corrected image; a segmentation unit configured to perform foreground-background segmentation on the to-be-corrected image to obtain background information of the to-be-corrected image; a prediction unit configured to perform backward prediction on the to-be-corrected image according to the deformation image feature and the background information to obtain a backward image of the to-be-corrected image; and a correction unit configured to correct the to-be-corrected image based on the backward image.

[0007] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected with the at least one processor; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the above method.

[0008] According to another aspect of the present disclosure, a non-transitory computer readable storage medium storing computer instructions is provided, the computer instructions being used to enable a computer to perform the above method.

[0009] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the above method.

[0010] According to one or more embodiments of the present disclosure, by introducing the background information of the to-be-corrected image, the contour of the foreground subject of the to-be-corrected image can be better determined when correcting the deformation of the to-be-corrected image based on the backward prediction method, so that the foreground subject and the background region in the to-be-corrected image can be accurately distinguished in the correction process, and the effect of image correction is improved. Moreover, the cooperation of such background information and backward prediction further solves the problem that the foreground segmentation result obtained by image segmentation cannot be applied to the backward image prediction task, and expands the application range.

[0011] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0012] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments and together with the description serve to explain exemplary implementations of the application. The illustrated embodiments are exemplary only and not limiting of the scope of the claims. In all the drawings, like reference numerals refer to like parts throughout the several views.

[0013] Figure 1 A schematic diagram illustrating an exemplary system in which various methods described herein can be implemented according to embodiments of the present disclosure is shown;

[0014] Figure 2 An exemplary flowchart of an image rectification method according to embodiments of the present disclosure is shown;

[0015] Figure 3 An exemplary partial flowchart of an image rectification method according to embodiments of the present disclosure is shown;

[0016] Figure 4 Another exemplary partial flowchart of an image rectification method according to embodiments of the present disclosure is shown;

[0017] Figure 5 Still another exemplary partial flowchart of an image rectification method according to embodiments of the present disclosure is shown;

[0018] Figure 6 Yet another exemplary partial flowchart of an image rectification method according to embodiments of the present disclosure is shown;

[0019] Figure 7 A structural block diagram of an image rectification apparatus according to embodiments of the present disclosure is shown; and

[0020] Figure 8 A structural block diagram of an exemplary electronic device that can be used to implement embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0021] Exemplary embodiments of the present disclosure are described herein with reference to the accompanying drawings, which are incorporated in this specification, wherein various details of the embodiments of the present disclosure are set forth in order to provide an overall understanding of the present disclosure. It should be understood that the various details of the embodiments of the present disclosure can be combined or eliminated in other embodiments of the present disclosure, and that such embodiments of the present disclosure will be understood and appreciated by those skilled in the art from the teachings of the present disclosure. Therefore, the following detailed description is not intended to limit the scope of the present disclosure, as claimed, but is merely illustrative of possible embodiments of the present disclosure. Accordingly, while the following is described in terms of specific embodiments, it will be appreciated that the embodiments of the present disclosure are not limited to those embodiments or the embodiments described.

[0022] In the present disclosure, the terms "first", "second", etc. are used to describe various elements, unless otherwise stated, and are not intended to imply a relative position, a temporal sequence, or an importance of the elements. In some examples, the first element and the second element can refer to the same instance of the element, and in some cases, based on the context of the description, they can also refer to different instances.

[0023] The terminology used in the description of the various described examples in the present disclosure is for the purpose of describing particular examples only and is not intended to be limiting. As used in this description, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. Furthermore, the use of the terms "a" and "an" and "the" and "at least one" are intended to include the meaning of "one or more." Additionally, the use of the term "including," as well as other forms such as "includes" and "included," is not limiting. Also, terms such as "example" and "exemplary" are utilized merely for illustrative purposes and do not reflect a partial or full scope of the subject matter.

[0024] Embodiments of the present disclosure will be described in detail with reference to the drawings, where like reference numerals can refer to like elements throughout the various figures.

[0025] Figure 1 A schematic diagram of an example system 100 in which various methods and apparatus described herein can be implemented in accordance with embodiments of the present disclosure is shown. Referring to Figure 1 , the system 100 includes one or more client devices 101, 102, 103, 104, 105, and 106, a server 120, and one or more communication networks 110 that couple the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105, and 106 can be configured to execute one or more application programs.

[0026] In embodiments of the present disclosure, the server 120 can run one or more services or software applications that enable the execution of an image rectification method.

[0027] In certain embodiments, the server 120 can also provide other services or software applications, which can include non-virtual environments and virtual environments. In certain embodiments, these services can be provided as web-based services or cloud services, such as to users of the client devices 101, 102, 103, 104, 105, and / or 106 under a software as a service (SaaS) network.

[0028] In Figure 1 In the illustrated configuration, the server 120 can include one or more components that implement the functionality performed by the server 120. These components can include software components that are executable by one or more processors, hardware components, or combinations thereof. Users operating the client devices 101, 102, 103, 104, 105, and / or 106 can in turn utilize one or more client application programs to interact with the server 120 to utilize the services provided by the components. It should be understood that various different system configurations are possible, which can differ from the system 100. Thus, Figure 1 is one example of a system for implementing the various methods described herein and is not intended to be limiting.

[0029] A user can use the client devices 101, 102, 103, 104, 105, and / or 106 to conduct the rectification of the morphed image. The client devices can provide an interface that enables a user of the client device to interact with the client device. The client devices can also output information to the user via the interface. Although Figure 1 Only six client devices are depicted, but one of skill in the art will appreciate that the present disclosure can support any number of client devices.

[0030] The client devices 101, 102, 103, 104, 105, and / or 106 can include various types of computer devices, such as portable handheld devices, general purpose computers (such as personal computers and laptop computers), workstation computers, wearable devices, smart screen devices, self-service kiosk devices, service robots, gaming systems, thin clients, various messaging devices, sensors or other sensing devices, and the like. These computer devices can run various types and versions of software applications and operating systems, such as MICROSOFT Windows, APPLE iOS, UNIX-like operating systems, Linux or Linux-like operating systems (such as GOOGLE Chrome OS); or include various mobile operating systems, such as MICROSOFT Windows Mobile OS, iOS, Windows Phone, Android. Portable handheld devices can include cellular telephones, smartphones, tablet computers, personal digital assistants (PDAs), and the like. Wearable devices can include head-mounted displays (such as smart glasses) and other devices. Gaming systems can include various handheld gaming devices, Internet-enabled gaming devices, and the like. The client devices are capable of executing various different applications, such as various Internet-related applications, communication applications (such as email applications), short message service (SMS) applications, and can use various communication protocols.

[0031] The networks 110 can be any type of networks known to those of skill in the art that can support data communications using any of a number of available protocols, including without limitation TCP / IP, SNA, IPX, etc. As examples only, one or more of the networks 110 can be a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a blockchain network, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, WIFI), and / or any combination of these and / or other networks.

[0032] The server 120 can include one or more general purpose computers, special purpose server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other appropriate arrangement and / or combination. The server 120 can include one or more virtual machines running virtual operating systems, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that can be virtualized to maintain virtual storage devices for the servers). In various embodiments, the server 120 can run one or more services or software applications that provide the functionality described below.

[0033] The computing units in the server 120 can run one or more operating systems including any of the operating systems described above, as well as any commercially available server operating systems. The server 120 can also run any of a variety of additional server applications and / or mid-tier applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0034] In some embodiments, the server 120 can include one or more applications to analyze and consolidate data feeds and / or event updates from users of the client devices 101, 102, 103, 104, 105, and 106. The server 120 can also include one or more applications to display the data feeds and / or real-time events via one or more display devices of the client devices 101, 102, 103, 104, 105, and 106.

[0035] In some embodiments, the server 120 can be a server of a distributed system, or a server combined with a blockchain. The server 120 can also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. The cloud server is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and virtual private server (VPS, Virtual Private Server) services.

[0036] The system 100 can also include one or more databases 130. In certain embodiments, these databases can be used to store data and other information. For example, one or more of the databases 130 can be used to store information such as image files. The databases 130 can reside at various locations. For example, databases used by the server 120 can be local to the server 120, or can be remote from the server 120 and can communicate with the server 120 via a network-based or dedicated connection. The databases 130 can be of different types. In certain embodiments, databases used by the server 120 can be, for example, relational databases. One or more of these databases can store, update, and retrieve data to and from the databases in response to commands.

[0037] In certain embodiments, one or more of the databases 130 can also be used by applications to store application data. Databases used by applications can be databases of different types, such as key-value stores, object stores, or regular stores supported by file systems.

[0038] Figure 1 The system 100 can be configured and operated in various ways to enable the application of various methods and apparatuses described in accordance with the present disclosure.

[0039] In the related art, a backward prediction method is generally used to correct a deformation image, however, the backward prediction method mainly performs coordinate prediction for the foreground subject of a distorted image, thus, for a deformation image with unclear or missing edge contour of the foreground subject, the backward prediction method has a problem of being unable to effectively determine the edge contour of the foreground subject in the deformation image, resulting in poor correction effect.

[0040] To this end, Figure 2 An exemplary flowchart of an image correction method according to an embodiment of the present disclosure is shown. The method can be implemented by a client or a server as shown in Figure 1 The method 200 in Figure 2 may be implemented by a client or a server as shown in

[0041] As shown in Figure 2 An image correction method 200 according to an embodiment of the present disclosure is provided, including: obtaining a to-be-corrected image (step 210); determining a deformation image feature of the to-be-corrected image, wherein the deformation image feature indicates a type and a degree of deformation of the to-be-corrected image (step 220); performing foreground-background segmentation on the to-be-corrected image to obtain background information of the to-be-corrected image (step 230); performing backward prediction on the to-be-corrected image according to the deformation image feature and the background information to obtain a backward graph of the to-be-corrected image (step 240); and correcting the to-be-corrected image based on the backward graph (step 250).

[0042] According to one or more embodiments of the present disclosure, by introducing the background information of the image to be corrected, the contour of the foreground subject of the image to be corrected can be better determined when the deformation of the image to be corrected is corrected based on the backward prediction method, so that the foreground subject and the background region in the image to be corrected can be accurately distinguished in the correction process, and the effect of the backward prediction method on the foreground subject image correction is improved. Moreover, the cooperation of such background information and backward prediction further solves the problem that the foreground segmentation result obtained by image segmentation cannot be applied to the backward prediction task, and expands the application range.

[0043] In step 210, an image to be corrected is obtained.

[0044] In some embodiments, the image to be corrected can be an image collected by a terminal device, or can be an image pre-stored in a database, which is not limited.

[0045] In step 220, a deformation image feature of the image to be corrected is determined, wherein the deformation image feature indicates the type and degree of deformation of the image to be corrected.

[0046] In some embodiments, the deformation described above can be generated in the image collection process, for example, due to the collection angle or movement of the terminal device during collection, resulting in distortion of the collected image; in other embodiments, the deformation described above can also be inherent to the foreground subject, for example, the image to be corrected is an image of a paper document collected, and the edges of the paper document are damaged or missing, resulting in uneven edges that cannot form a rectangular document area.

[0047] It should be noted that the deformation type of the image to be corrected described above is only for illustrative purposes and is not limited thereto.

[0048] Figure 3 An exemplary partial flowchart of an image correction method according to an embodiment of the present disclosure is shown.

[0049] According to some embodiments, as shown in Figure 3 Step 220 includes: performing feature extraction on the image to be corrected to obtain a plurality of local image features (step 321); calculating the feature similarity between each two local image features in the plurality of local image features to obtain a plurality of feature similarities (step 322); and generating a deformation image feature based on the plurality of local image features and the plurality of feature similarities (step 323).

[0050] By determining the global deformation image feature of the image to be corrected through the similarity of the local features of the image to be corrected, the global deformation information of the image to be corrected can be more effectively represented to improve the final correction efficiency and correction effect.

[0051] At step 321, feature extraction is performed on the image to be corrected to obtain a plurality of local image features.

[0052] In some embodiments, the first convolutional neural network can be used to process the image to be corrected to down-sample the image to be corrected, thereby obtaining local image features of the image to be corrected. The local image features can be shallow image texture features or deep image semantic features, which are not limited.

[0053] In an example, the first convolutional neural network can include 7 convolutional layers to down-sample the resolution of the image to be corrected to 1 / 8 of the original image. It should be noted that the number of convolutional layers and the sampling rate of down-sampling are only used as examples and are not limited thereto.

[0054] In some embodiments, the image to be corrected can be divided into a plurality of sub-regions, and feature extraction is performed on each sub-region to obtain local image features corresponding to the sub-region, thereby obtaining a plurality of local image features. It should be understood that the present disclosure does not limit the way the image to be corrected is divided into sub-regions and the number of sub-regions obtained.

[0055] At step 322, the feature similarity between each two local image features in the plurality of local image features is calculated to obtain a plurality of feature similarities.

[0056] At step 323, a deformation image feature is generated based on the plurality of local image features and the plurality of feature similarities.

[0057] In some embodiments, the image correction method of the embodiments of the present disclosure can be implemented based on a deep learning model, which includes but is not limited to a Transformer model.

[0058] In an example, the Transformer model can include 6 encoders and corresponding 6 decoders, each encoder including a self-attention layer, a fully connected layer, and a residual network structure, and each encoder including a self-attention layer, a cross-attention layer, a fully connected layer, and a residual network structure. It should be understood that the above description of the structure of the Transformer model is only used as an example and is not limited thereto.

[0059] In an example, by using the 6 decoders of the Transformer model to process the plurality of local image features obtained at step 321, the similarity between the self-attention features corresponding to the plurality of local image features and the learnable vectors can be calculated, and the deformation image feature with global features can be obtained by weighted summation through the self-attention mechanism.

[0060] At step 230, foreground and background segmentation is performed on the image to be corrected to obtain background information of the image to be corrected.

[0061] Foreground and background segmentation can be achieved based on image segmentation models. However, conventional image segmentation models typically output the segmented foreground information. To obtain the background information obtained from image segmentation and to cooperate with backward prediction methods for image correction, Figure 4 A flowchart of another exemplary portion of an image correction method according to an embodiment of the present disclosure is shown.

[0062] According to some embodiments, such as Figure 4 As shown, step 230 includes: determining the probability that each pixel in the image to be corrected belongs to the foreground (step 431); taking pixels in the image to be corrected with a probability less than a preset threshold as background pixels to obtain multiple background pixels (step 432); and generating background information based on the multiple background pixels (step 433).

[0063] In some embodiments, a pre-trained image segmentation model can be used to implement steps 431 to 433 above. The pre-trained image segmentation model is trained to output background information obtained by segmenting the image.

[0064] In some embodiments, the pre-trained image segmentation model described above can be used together with the Transformer model as a whole image correction model, or it can be used as an independent model to output background information so that the Transformer model can perform image correction based on the backward prediction method and the background information. There is no limitation on this.

[0065] In some embodiments, the input to the pre-trained image segmentation model described above can be the image to be corrected, or it can be the image deformation features extracted based on the image to be corrected, without any limitation.

[0066] In some embodiments, the image segmentation model described above uses a binary cross-entropy loss function for network supervision loss during pre-training to achieve constrained convergence, as expressed in the following formula:

[0067]

[0068] Among them, L seg The loss function of the image segmentation model is represented by ; N represents the number of pixels in the sample images used for training; i represents the i-th pixel among the N pixels; y i This indicates the labeling done during training to determine whether the i-th pixel belongs to the foreground or background. i ∈[0,1];p i This represents the probability that the i-th pixel, predicted by the image segmentation model, belongs to the foreground.

[0069] At step 240, the warped image features and the background information are used to perform backward prediction on the image to be corrected to obtain a backward map of the image to be corrected.

[0070] According to some embodiments, the method 200 further includes performing a flattening operation on the warped image features to make the number of dimensions of the warped image features reach a preset number of dimensions. Moreover, the step 240 includes performing backward prediction on the image to be corrected according to the warped image features subjected to the flattening operation and the background information to obtain a backward map of the image to be corrected.

[0071] Different network models have different requirements for the dimensions of input features. Based on this, the warped image features are reduced in dimension through the flattening operation, so that they can meet the requirements of being input to different models to implement image correction based on the models.

[0072] Figure 5 Another exemplary partial flowchart of an image correction method according to an embodiment of the present disclosure is shown.

[0073] According to some embodiments, as shown in Figure 5 The step 240 includes predicting a coordinate offset of each pixel in the image to be corrected based on the warped image features and the background information (step 541), and generating a backward map based on the coordinate offsets of a plurality of pixels in the image to be corrected (step 542).

[0074] In some embodiments, the decoder of the Transformer model can decode the information required for correction contained in each feature vector from the warped image features output by the encoder. The information required for correction is processed by the backward prediction model, so that the coordinate offset of each pixel in the predicted corrected image from the corresponding pixel in the image to be corrected can be output to generate a backward map.

[0075] In some embodiments, the backward prediction model can be used together with the Transformer model as an integrated image correction model, or can be used as an independent model to output a backward map, and no limitation is made in this regard.

[0076] In some embodiments, the backward prediction model uses an L1 loss function to directly constrain the difference between the coordinate offset output by the model and the real offset during training to perform network supervision loss, so as to achieve constraint convergence, which is expressed by the following formula:

[0077] L dewarp =‖f gt -f b ‖

[0078] wherein L dewarp represents the loss function of the backward prediction model, fgt f represents the true label obtained by ground truth labeling of the backward graph of the samples used as training data. b This represents the backward graph output by the backward prediction model.

[0079] Figure 6 A further exemplary flowchart of an image correction method according to an embodiment of the present disclosure is shown.

[0080] According to some embodiments, such as Figure 6 As shown, when the deformed image features have been flattened, step 240 includes: predicting the correction coordinates of each pixel in the image to be corrected based on the flattened deformed image features and background information to obtain the predicted coordinate offset of the pixel (step 641); and generating a backward map based on the multiple predicted coordinate offsets corresponding to multiple pixels in the image to be corrected (step 642).

[0081] For a detailed explanation of steps 641 and 642, please refer to steps 541 and 542 above, which will not be repeated here.

[0082] Figure 7 A structural block diagram of an image correction apparatus according to an embodiment of the present disclosure is shown.

[0083] like Figure 7 As shown, according to an embodiment of this disclosure, an image correction apparatus 700 is provided, comprising: an acquisition unit 710 configured to acquire an image to be corrected; a determination unit 720 configured to determine deformation image features of the image to be corrected, wherein the deformation image features indicate the type and degree of deformation occurring in the image to be corrected; a segmentation unit 730 configured to perform foreground-background segmentation on the image to be corrected to obtain background information of the image to be corrected; a prediction unit 740 configured to perform backward prediction on the image to be corrected based on the deformation image features and the background information to obtain a backward image of the image to be corrected; and a correction unit 750 configured to correct the image to be corrected based on the backward image.

[0084] According to some embodiments, the determining unit 720 includes: an extraction subunit 721 configured to extract features from the image to be corrected to obtain multiple local image features; a calculation subunit 722 configured to calculate the feature similarity between every two local image features to obtain multiple feature similarities; and a feature subunit 723 configured to generate deformed image features based on the multiple local image features and the multiple feature similarities.

[0085] According to some embodiments, the segmentation unit 730 comprises: a probability subunit 731 configured to determine a probability that each pixel point in the image to be corrected belongs to the foreground; a background subunit 732 configured to take the pixel points in the image to be corrected with probabilities less than a preset threshold as background pixel points to obtain a plurality of background pixel points; and a generation subunit 733 configured to generate the background information based on the plurality of background pixel points.

[0086] According to some embodiments, the image correction apparatus 700 further comprises: a flattening unit 760 configured to perform a flattening operation on the morphed image feature to make the dimension number of the morphed image feature reach a preset dimension number, wherein the prediction unit 740 is further configured to perform backward prediction on the image to be corrected according to the morphed image feature after the flattening operation and the background information to obtain a backward image of the image to be corrected.

[0087] According to some embodiments, the prediction unit 740 comprises: a first prediction subunit 741 configured to perform correction coordinate prediction on each pixel point in the image to be corrected based on the morphed image feature to obtain a predicted coordinate offset of the pixel point; and a first backward image subunit 742 configured to generate a backward image based on a plurality of predicted coordinate offsets corresponding to a plurality of pixel points in the image to be corrected.

[0088] According to some embodiments, the prediction unit 740 comprises: a second prediction subunit 743 configured to perform correction coordinate prediction on each pixel point in the image to be corrected based on the morphed image feature after the flattening operation to obtain a predicted coordinate offset of the pixel point; and a second backward image subunit 744 configured to generate a backward image based on a plurality of predicted coordinate offsets corresponding to a plurality of pixel points in the image to be corrected.

[0089] Here, the operations of the above-mentioned units 710-750 of the image correction apparatus 700 are similar to the operations of the steps 210-250 described above, and will not be described here again.

[0090] According to embodiments of the present disclosure, an electronic device, a readable storage medium and a computer program product are also provided.

[0091] Reference Figure 8The present invention describes a structural block diagram of an electronic device 800 that can serve as a server or client of the present disclosure, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0092] like Figure 8 As shown, the electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. The RAM 803 may also store various programs and data required for the operation of the electronic device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0093] Multiple components in electronic device 800 are connected to I / O interface 805, including: input unit 806, output unit 807, storage unit 808, and communication unit 809. Input unit 806 can be any type of device capable of inputting information to electronic device 800. Input unit 806 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of electronic device, and can include, but is not limited to, a mouse, keyboard, touchscreen, trackpad, trackball, joystick, microphone, and / or remote control. Output unit 807 can be any type of device capable of presenting information, and can include, but is not limited to, a monitor, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 808 can include, but is not limited to, disk and optical disk. Communication unit 809 allows electronic device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and can include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and / or chipsets, such as Bluetooth devices, 802.11 devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0094] The computing unit 801 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning network algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs various methods and processes described above, such as the method 200. For example, in some embodiments, the method 200 can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded onto the RAM 803 and executed by the computing unit 801, one or more steps of the method 200 described above can be performed. Alternatively, in other embodiments, the computing unit 801 can be configured to perform the method 200 by any other suitable means, such as by means of firmware.

[0095] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a complex programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0096] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or the block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0097] In the context of this disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0098] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0099] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0100] The computer system can include clients and servers. This relationship can be remote or on-site. The servers can be cloud servers, servers of a distributed system, or servers combined with a blockchain.

[0101] It should be understood that the various forms of flow shown above can be used to reorder, add, or delete steps. For example, the steps described in the present disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved, which are not limited herein.

[0102] Although embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-described methods, systems, and devices are merely exemplary embodiments or examples, and the scope of the present disclosure is not limited by these embodiments or examples, but is only limited by the granted claims and their equivalent scope. Various elements in the embodiments or examples can be omitted or replaced by equivalent elements. In addition, each step can be performed in an order different from that described in the present disclosure. Further, various elements in the embodiments or examples can be combined in various ways. It is important that many of the elements described herein can be replaced by equivalent elements that appear after the present disclosure as technology evolves.

Claims

1. A method for image rectification, comprising: obtaining an image to be rectified; determining a deformation image feature of the image to be rectified, wherein the deformation image feature indicates a type and a degree of deformation of the image to be rectified; performing foreground-background segmentation on the image to be rectified to obtain background information of the image to be rectified; performing backward prediction on the image to be rectified according to the deformation image feature and the background information to obtain a backward map of the image to be rectified, wherein the backward prediction comprises: performing rectification coordinate prediction on each pixel point in the image to be rectified based on the deformation image feature and the background information to obtain a predicted coordinate offset of the pixel point; and generating the backward map based on a plurality of predicted coordinate offsets corresponding to a plurality of pixel points in the image to be rectified; and rectifying the image to be rectified based on the backward map.

2. The method of claim 1, wherein, The determining of the deformation image feature of the image to be rectified comprises: performing feature extraction on the image to be rectified to obtain a plurality of local image features; calculating feature similarities between each two local image features in the plurality of local image features to obtain a plurality of feature similarities; and generating the deformation image feature based on the plurality of local image features and the plurality of feature similarities.

3. The method of claim 1, wherein, The foreground-background segmentation on the image to be rectified comprises: determining a probability that each pixel point in the image to be rectified belongs to a foreground; regarding pixel points in the image to be rectified with a probability less than a preset threshold as background pixel points to obtain a plurality of background pixel points; and generating the background information based on the plurality of background pixel points.

4. The method of claim 1, further comprising: performing a flattening operation on the deformation image feature to make a dimension number of the deformation image feature reach a preset dimension number, wherein the backward prediction on the image to be rectified according to the deformation image feature and the background information to obtain the backward map of the image to be rectified comprises: performing the backward prediction on the image to be rectified according to the deformation image feature subjected to the flattening operation and the background information to obtain the backward map of the image to be rectified.

5. The method of claim 4, wherein, The backward prediction on the image to be rectified according to the deformation image feature subjected to the flattening operation and the background information to obtain the backward map of the image to be rectified comprises: performing rectification coordinate prediction on each pixel point in the image to be rectified based on the deformation image feature subjected to the flattening operation and the background information to obtain a predicted coordinate offset of the pixel point; and generating the backward map based on a plurality of predicted coordinate offsets corresponding to a plurality of pixel points in the image to be rectified.

6. An apparatus for image rectification, comprising: an obtaining unit configured to obtain an image to be rectified; a determining unit configured to determine a deformation image feature of the image to be rectified, wherein the deformation image feature indicates a type and a degree of deformation of the image to be rectified; a segmentation unit configured to perform foreground-background segmentation on the image to be rectified to obtain background information of the image to be rectified; a prediction unit, configured to perform backward prediction on the image to be corrected according to the morphological image feature and the background information to obtain a backward map of the image to be corrected, wherein the prediction unit comprises: a first prediction subunit, configured to perform coordinate correction prediction on each pixel point in the image to be corrected based on the morphological image feature to obtain a predicted coordinate offset of the pixel point; and a first backward map subunit, configured to generate the backward map based on a plurality of predicted coordinate offsets corresponding to a plurality of pixel points in the image to be corrected; and a correction unit, configured to correct the image to be corrected based on the backward map.

7. The apparatus of claim 6, wherein, The determination unit comprises: an extraction subunit, configured to perform feature extraction on the image to be corrected to obtain a plurality of local image features; a calculation subunit, configured to calculate feature similarity between each two local image features in the plurality of local image features to obtain a plurality of feature similarities; and a feature subunit, configured to generate the morphological image feature based on the plurality of local image features and the plurality of feature similarities.

8. The apparatus of claim 6, wherein, The segmentation unit comprises: a probability subunit, configured to determine a probability that each pixel point in the image to be corrected belongs to the foreground; a background subunit, configured to take pixel points in the image to be corrected with a probability less than a preset threshold as background pixel points to obtain a plurality of background pixel points; and a generation subunit, configured to generate the background information based on the plurality of background pixel points.

9. The apparatus according to claim 6, further comprising: a flattening unit, configured to perform a flattening operation on the morphological image feature to make a dimension number of the morphological image feature reach a preset dimension number; wherein the prediction unit is further configured to: perform backward prediction on the image to be corrected according to the morphological image feature subjected to the flattening operation and the background information to obtain a backward map of the image to be corrected.

10. The apparatus of claim 9, wherein, The prediction unit comprises: a second prediction subunit, configured to perform coordinate correction prediction on each pixel point in the image to be corrected based on the morphological image feature subjected to the flattening operation to obtain a predicted coordinate offset of the pixel point; and a second backward map subunit, configured to generate the backward map based on a plurality of predicted coordinate offsets corresponding to a plurality of pixel points in the image to be corrected.

11. An electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-5.

12. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to enable the computer to perform the method of any one of claims 1-5.

13. A computer program product comprising a computer program, wherein, The computer program, when executed by a processor, implements the method of any one of claims 1-5.

Images