Plane correction method and device, computer readable medium and electronic device

Abstract

A plane correction method, a plane correction device, a computer-readable medium and an electronic device, relating to the field of image processing technology. The method comprises: obtaining plane parameters of a target plane in an image to be processed, calculating a position identification line and a correction matrix corresponding to the image to be processed according to the plane parameters; when the position identification line and the border of the image to be processed have an intersection on the image plane, determining a correction area in the image to be processed according to the plane parameters, and correcting the correction area according to the correction matrix to obtain a corrected image. This can avoid the problem in the related art that when the image to be processed is directly corrected by the correction matrix, the correction result is distorted because the image to be processed contains too much non-target plane information.

Description

Background Art

[0002] The Manhattan World Hypothesis holds that most man-made objects, such as buildings and roads, have strong orthogonal properties. There exists a rectangular coordinate system that makes the surface of the object parallel or perpendicular to the coordinate axis, such as Figure 1 As shown. When photographing and scanning the surface of an object in the three-dimensional world, this hypothesis can be used to construct a plane model, and the image to be processed can be processed by plane detection and correction, so that the image to be processed is converted into a corrected image containing a front view of the object surface, so as to obtain information about the object surface. For example, when scanning a paper document, the image to be processed can be processed by plane detection and correction, so that the image to be processed is converted into a corrected image containing a front view of the paper document, so as to read text and other information in the paper document according to the corrected image. Summary of the invention

[0003] The purpose of the present disclosure is to provide a plane correction method, a plane correction device, a computer-readable medium and an electronic device, so as to at least to some extent avoid the problem in the related art that when a to-be-processed image is corrected directly through a correction matrix, the to-be-processed image contains too much non-target plane information, resulting in distortion of the correction result.

[0004] According to a first aspect of the present disclosure, a plane correction method is provided, comprising: obtaining plane parameters of a target plane in an image to be processed, and calculating a position identification line and a correction matrix corresponding to the image to be processed according to the plane parameters; wherein the position identification line is on an image plane of the image to be processed; when an intersection exists between the position identification line and a border of the image to be processed on the image plane, determining a correction area in the image to be processed according to the plane parameters, and performing correction on the correction area according to the correction matrix to obtain a corrected image.

[0005] According to a second aspect of the present disclosure, a plane correction device is provided, including: a parameter calculation module, used to obtain plane parameters of a target plane in an image to be processed, and calculate a position identification line and a correction matrix corresponding to the image to be processed according to the plane parameters; wherein the position identification line is on the image plane of the image to be processed; and a plane correction module, used to determine a correction area in the image to be processed according to the plane parameters when the position identification line and the border of the image to be processed have an intersection on the image plane, and perform correction on the correction area according to the correction matrix to obtain a corrected image.

[0006] According to a third aspect of the present disclosure, a computer-readable medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the above method is implemented.

[0007] According to a fourth aspect of the present disclosure, there is provided an electronic device, characterized in that it includes:

[0008] Processor; and

[0009] The memory is used to store one or more programs. When the one or more programs are executed by one or more processors, the one or more processors implement the above method.

[0010] A plane correction method provided by an embodiment of the present disclosure obtains the plane parameters of the target plane in the image to be processed, and calculates the position identification line and correction matrix corresponding to the image to be processed according to the plane parameters; then, when the position identification line and the border of the image to be processed have an intersection on the image plane, the correction area is determined in the image to be processed according to the plane parameters, and the correction area is corrected to obtain a corrected image. On the one hand, the position identification line can be used to determine whether errors will occur when the image to be processed is directly corrected through the correction matrix; on the other hand, when the position identification line and the border of the image to be processed have an intersection on the image plane, the correction area can be determined in the image to be processed according to the plane parameters, and then the correction area is corrected through the correction matrix, thereby avoiding the problem of errors in the correction result due to the image to be processed containing too much non-target plane information.

[0011] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0013] Figure 1 A schematic diagram showing the orthogonal properties of most objects in a three-dimensional control;

[0014] Figure 2 A schematic diagram showing an exemplary system architecture to which embodiments of the present disclosure may be applied;

[0015] Figure 3 A schematic diagram showing an electronic device to which the embodiments of the present disclosure can be applied;

[0016] Figure 4 An image to be processed is shown;

[0017] Figure 5 It shows that the related technology Figure 4 Correct the image to be processed to obtain a corrected image;

[0018] Figure 6 Another image to be processed and its corresponding rectified image are shown;

[0019] Figure 7 Schematically showing another image to be processed and its corresponding rectified image;

[0020] Figure 8 Schematically showing another image to be processed and its corresponding corrected image;

[0021] Fig. 9 A flow chart schematically illustrates a method for plane correction in an exemplary embodiment of the present disclosure;

[0022] Fig.10 A schematic diagram of a principle for determining a position identification line in an exemplary embodiment of the present disclosure is shown;

[0023] Fig.11 Schematically showing another principle diagram for determining a position identification line in an exemplary embodiment of the present disclosure;

[0024] Fig.12 A schematic diagram schematically illustrates a border enhancement method in an exemplary embodiment of the present disclosure;

[0025] Fig.13 A schematic diagram schematically illustrates another border enhancement method in an exemplary embodiment of the present disclosure;

[0026] Fig.14 A schematic diagram schematically illustrates a method for determining a correction area in an exemplary embodiment of the present disclosure;

[0027] Fig.15 A schematic diagram schematically illustrates another method for determining a correction area in an exemplary embodiment of the present disclosure;

[0028] Fig.16 A schematic diagram schematically illustrates another method for determining a correction area in an exemplary embodiment of the present disclosure;

[0029] Fig.17 A schematic diagram schematically illustrates a method of offsetting a position identification line in an exemplary embodiment of the present disclosure;

[0030] Fig.18 A flow chart schematically illustrates another plane correction method in an exemplary embodiment of the present disclosure;

[0031] Fig.19 A schematic diagram schematically shows the composition of a planar correction device in an exemplary embodiment of the present disclosure;

[0032] Fig. 20 The figure schematically shows the composition of another planar correction device in an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0033] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0034] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0035] Figure 2 A schematic diagram of a system architecture of an exemplary application environment in which a plane correction method and apparatus according to an embodiment of the present disclosure can be applied is shown.

[0036] like Figure 2 As shown, the system architecture 200 may include one or more of terminal devices 201, 202, 203, a network 204, and a server 205. The network 204 is used to provide a medium for communication links between the terminal devices 201, 202, 203 and the server 205. The network 204 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc. The terminal devices 201, 202, 203 may be various electronic devices with image processing functions, including but not limited to desktop computers, portable computers, smart phones, tablet computers, etc. It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. According to the implementation requirements, there may be any number of terminal devices, networks and servers. For example, the server 205 may be a server cluster composed of multiple servers.

[0037] The plane correction method provided in the embodiment of the present disclosure is generally executed by the terminal devices 201, 202, and 203, and accordingly, the plane correction device is generally disposed in the terminal devices 201, 202, and 203. However, it is easy for those skilled in the art to understand that the plane correction method provided in the embodiment of the present disclosure can also be executed by the server 205, and accordingly, the plane correction device can also be disposed in the server 205, which is not particularly limited in this exemplary embodiment. For example, in an exemplary embodiment, a user may obtain the plane parameters of the target plane in the image to be processed through the terminal devices 201, 202, and 203, calculate the position identification line and the correction matrix, and when the position identification line and the border of the image to be processed have an intersection on the image plane, determine the correction area in the image to be processed according to the plane parameters, and then correct the correction area according to the calculated correction matrix to obtain the corrected image corresponding to the target plane; in another embodiment, the server 205 may obtain the relevant data provided by the terminal devices 201, 202, and 203, process the data through the plane correction method, and then feed back the processing result, i.e., the corrected image, to the terminal devices 201, 202, and 203, etc.

[0038] An exemplary embodiment of the present disclosure provides an electronic device for implementing a planar correction method, which may be Figure 1 The terminal device 201, 202, 203 or the server 205 in the embodiment of the present invention comprises at least a processor and a memory, the memory is used to store executable instructions of the processor, and the processor is configured to execute the plane correction method by executing the executable instructions.

[0039] Below Figure 3 Taking the mobile terminal 300 in FIG. 1 as an example, the structure of the electronic device is exemplarily described. It should be understood by those skilled in the art that, in addition to the components specifically used for mobile purposes, Figure 3 The structure in the figure can also be applied to fixed type devices. In other embodiments, the mobile terminal 300 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure can be implemented in hardware, software, or a combination of software and hardware. The interface connection relationship between the components is only shown schematically and does not constitute a structural limitation on the mobile terminal 300. In other embodiments, the mobile terminal 300 may also adopt the same Figure 3 Different interface connection methods, or a combination of multiple interface connection methods.

[0040] like Figure 3As shown, the mobile terminal 300 may specifically include: a processor 310, an internal memory 321, an external memory interface 322, a Universal Serial Bus (USB) interface 330, a charging management module 340, a power management module 341, a battery 342, an antenna 1, an antenna 2, a mobile communication module 350, a wireless communication module 360, an audio module 370, a speaker 371, a receiver 372, a microphone 373, an earphone interface 374, a sensor module 380, a display screen 390, a camera module 391, an indicator 392, a motor 393, a button 394, and a subscriber identification module (SIM) card interface 395, etc. The sensor module 380 may include a depth sensor 3801, a pressure sensor 3802, a gyroscope sensor 3803, etc.

[0041] The processor 310 may include one or more processing units, for example, the processor 310 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor and / or a neural network processor (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0042] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of the mobile terminal 300 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0043] The processor 310 is provided with a memory which can store instructions for implementing six modular functions: detection instructions, connection instructions, information management instructions, analysis instructions, data transmission instructions and notification instructions, and the execution is controlled by the processor 310.

[0044] The charging management module 340 is used to receive charging input from the charger. The power management module 341 is used to connect the battery 342, the charging management module 340 and the processor 310. The power management module 341 receives input from the battery 342 and / or the charging management module 340 to power the processor 310, the internal memory 321, the display screen 390, the camera module 391 and the wireless communication module 360.

[0045] The wireless communication function of the mobile terminal 300 can be implemented through antenna 1, antenna 2, mobile communication module 350, wireless communication module 360, modulation and demodulation processor and baseband processor. Among them, antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals; mobile communication module 350 can provide solutions for wireless communication including 2G / 3G / 4G / 5G applied on mobile terminal 300; modulation and demodulation processor can include modulator and demodulator; wireless communication module 360 ​​can provide solutions for wireless communication including wireless local area network (WLAN) (such as wireless fidelity (Wi-Fi) network), Bluetooth (BT) and the like applied on mobile terminal 300. In some embodiments, antenna 1 of mobile terminal 300 is coupled with mobile communication module 350, and antenna 2 is coupled with wireless communication module 360, so that mobile terminal 300 can communicate with network and other devices through wireless communication technology.

[0046] The mobile terminal 300 implements the display function through a GPU, a display screen 390, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 390 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 310 may include one or more GPUs that execute program instructions to generate or change display information.

[0047] The mobile terminal 300 can realize the shooting function through the ISP, the camera module 391, the video codec, the GPU, the display screen 390 and the application processor, etc. Among them, the ISP is used to process the data fed back by the camera module 391; the camera module 391 is used to capture static images or videos; the digital signal processor is used to process digital signals, and can process other digital signals in addition to digital image signals; the video codec is used to compress or decompress digital videos, and the mobile terminal 300 can also support one or more video codecs.

[0048] The external memory interface 322 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the mobile terminal 300. The external memory card communicates with the processor 310 through the external memory interface 322 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0049] The internal memory 321 can be used to store computer executable program codes, and the executable program codes include instructions. The internal memory 321 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the mobile terminal 300 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 321 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash memory (Universal Flash Storage, UFS), etc. The processor 310 executes various functional applications and data processing of the mobile terminal 300 by running instructions stored in the internal memory 321 and / or instructions stored in a memory provided in the processor.

[0050] The mobile terminal 300 can implement audio functions such as music playing and recording through the audio module 370 , the speaker 371 , the receiver 372 , the microphone 373 , the earphone interface 374 and the application processor.

[0051] The depth sensor 3801 is used to obtain the depth information of the scene. In some embodiments, the depth sensor can be set in the camera module 391, and the camera module 391 and the depth sensor 3801 can simultaneously collect data of the scene to obtain the color data and depth data of the image to be processed, that is, the point cloud data corresponding to the image to be processed.

[0052] The pressure sensor 3802 is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 3802 can be arranged on the display screen 390. There are many types of pressure sensors 3802, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc.

[0053] The gyro sensor 3803 can be used to determine the motion posture of the mobile terminal 300. In some embodiments, the angular velocity of the mobile terminal 300 around three axes (i.e., x, y, and z axes) can be determined by the gyro sensor 3803. The gyro sensor 3803 can be used for shooting anti-shake, navigation, somatosensory game scenes, etc.

[0054] In addition, sensors with other functions may be set in the sensor module 380 according to actual needs, such as an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, etc.

[0055] The mobile terminal 300 may also include other devices that provide auxiliary functions. For example, the key 394 includes a power key, a volume key, etc., and the user can input through the key to generate key signal input related to the user settings and function control of the mobile terminal 300. Another example is the indicator 392, the motor 393, the SIM card interface 395, etc.

[0056] In related technologies, in order to correct the target plane in the image to be processed, developers directly correct the image to be processed using the correction matrix obtained by plane parameter calculation. However, when correcting in this way, the correction result may be distorted because the image to be processed contains other information besides the target plane. For example, Figure 4 There is a target plane 410 in the image to be processed as shown. If the image to be processed is rectified directly, the following may be obtained: Figure 5 In the corrected image shown, the target plane 410 is corrected to the target plane 510, which is obviously different from the user's requirement; for example, Figure 6 The correction process shown also showed obvious errors.

[0057] The above distortion is mainly caused by the presence of a large amount of information outside the target plane in the image to be processed. Correcting the target plane in the two-dimensional image to be processed is actually a perspective transformation of the target plane in the two-dimensional image to be processed, and this perspective transformation is not applicable to points beyond the vanishing line range corresponding to the target plane. If points beyond the vanishing line range appear in the correction range, meaningless points will appear in the corrected image, so correction errors are likely to occur. Among them, the vanishing line refers to the image formed in the camera when the plane extends to infinity, and the vanishing line range refers to the image formed in the camera when the plane is less than infinity.

[0058] In order to solve the above problems, the following two methods are usually adopted: one is to use the boundary information of the image to be processed to determine the range of the target plane of the image to be processed, and then correct the range; the other is to crop the image to be processed and correct the cropped part.

[0059] However, in the first method, if the boundary information in the image to be processed is insufficient or the boundary is unclear, it is likely that the range of the target plane cannot be accurately determined, which will lead to errors in the correction; in the second method, since cropping is usually based on the image information of the image to be processed, the cropped target plane may not meet the user's requirements. In this case, some parts outside the target plane need to be corrected so that the user can manually correct the corrected image. Since the parts outside the target plane need to be corrected during correction, there is still the possibility of errors in the correction.

[0060] In the second method, refer to Figure 7 As shown in the figure, after the correction area including the target plane is corrected, the target plane may be corrected to a very small part of the corrected image. In this case, even if manual cropping is performed, since the resolution of the correction area in the image to be processed is limited, the resolution of the corrected image is also limited. Therefore, when the corrected image is manually corrected and the area corresponding to the target plane in the corrected image is selected for cropping, the corrected image corresponding to the target plane is likely to have a low image quality problem; refer to Figure 8 As shown, in the correction area including the target plane, there may still be points beyond the range of the vanishing line, so correction errors may occur, causing the corrected image to be distorted.

[0061] Based on the above problems, the present disclosure provides a plane correction method and a plane correction device, which are described in detail below.

[0062] Fig. 9 The flow chart of a plane correction method in this exemplary embodiment includes the following steps S910 and S920:

[0063] In step S910, the plane parameters of the target plane in the image to be processed are obtained, and the position identification line and the correction matrix corresponding to the image to be processed are calculated according to the plane parameters.

[0064] In an exemplary embodiment, the above-mentioned plane parameters are algebraic representation parameters of the target plane in three-dimensional space, which are used to determine a unique infinite plane in three-dimensional space, for example, P(a, b, c, d); the correction matrix is ​​a matrix calculated based on the plane parameters and used to perform perspective transformation on a two-dimensional image and correct the image. It should be noted that although the target plane in the image to be processed is a finite plane, when describing the target plane, the algebraic representation parameters in three-dimensional space represent an infinite plane in space, that is, P(a, b, c, d) represents a fixed infinite plane in space.

[0065] Among them, the position identification line is a specific line or surface in the three-dimensional space. The imaging on the image plane where the image to be processed is located can identify the specific position in the three-dimensional space in the two-dimensional image to be processed based on the perspective transformation principle, and then divide the image plane into near scene imaging that is closer than the specific line or surface, and far scene imaging that is farther than the specific line or surface.

[0066] It should be noted that the position identification line is always located on the image plane corresponding to the image to be processed, which is the imaging plane of the camera. Theoretically, the image plane is an infinite plane and is not limited by the size of the camera's photosensitive element.

[0067] In an exemplary embodiment, the camera can collect point cloud data corresponding to the image to be processed by means of a photosensitive element and a depth sensor. Therefore, the plane parameters of the target plane in the image to be processed can be obtained by performing plane detection on the point cloud data. The plane detection process can be performed by methods such as RANSAC, and the present disclosure does not specifically limit this.

[0068] It should be noted that the image to be processed may include multiple planes at the same time, so it is necessary to first determine the target plane among the planes to be processed. Usually, when it is necessary to obtain information about the surface of an object, the area occupied by the surface of the object in the corresponding image to be processed is the largest. Therefore, plane detection can be performed on the point cloud data of the image to be processed, and the plane that occupies the largest area on the image to be processed among all the planes that can be detected can be determined as the target plane, and then the plane parameters corresponding to the target plane can be obtained.

[0069] In an exemplary embodiment, due to the perspective transformation of the target plane, errors may occur after transforming points outside the range of the corresponding vanishing line of the target plane. Therefore, the vanishing line corresponding to the target plane can be used as a position identification line to ensure that the perspective transformation does not cause errors.

[0070] For example, the camera intrinsic parameters corresponding to the camera that captured the image to be processed can be obtained, and then the vanishing line can be calculated based on the camera intrinsic parameters and the plane parameters, and the vanishing line can be used as the position identification line. The vanishing line refers to the imaging result on the image plane when the target plane extends to infinity. In the specific calculation, the normal vector n of the target plane can be calculated by the plane parameters, and then the vanishing line can be calculated according to the following formula (1):

[0071] l 1 =K-1n Formula (1)

[0072] Among them, l 1 is the vanishing line, K is the camera intrinsic parameter corresponding to the camera that shoots the image to be processed, and n is the normal vector of the target plane.

[0073] In another exemplary embodiment, a preset plane can be set, and the position identification line can be determined according to the plane parameters of the target plane and the preset plane. The preset plane is parallel to the image plane of the camera, and the distance between the preset plane and the image plane is a preset distance. Specifically, the intersection line of the target plane and the preset plane in three-dimensional space can be determined according to the plane parameters, and the image of the intersection line in the image plane corresponding to the image to be processed is calculated, and the image is used as the position identification line.

[0074] It should be noted that the actual distance that the camera's depth sensor can effectively measure is limited. Therefore, in order to obtain effective information about the target plane, the distance between the target plane and the camera should usually be less than the distance that the depth sensor can effectively measure. On the contrary, if the target plane is outside the effective measurable distance, the depth data obtained is likely to be erroneous, and therefore the surface information of the target plane cannot be obtained. For the above reasons, through the different properties of the camera's own depth sensor, preset planes of different distances can be set to facilitate the determination of the position identification line.

[0075] For example, when scanning a paper document, the paper document needs to be placed within the distance range that the camera can effectively measure. Assuming that the measurement range of the depth sensor of a certain camera is within 2 meters, at this time, the paper document needs to be placed within a range of 2 meters, and the distance between the preset plane and the image plane of the image to be processed is set to 2 meters. The preset plane can be represented by an algebraic representation parameter in three-dimensional space, for example, Q(0, 0, 1, -2). In this case, the preset plane prevents the target plane from extending to infinity, so the imaging of the above-mentioned intersection on the image plane can further reduce the area within the vanishing line range to ensure that there are no points in the imaging of close scenes in the image plane that are closer than the intersection distance that will cause errors in the perspective transformation.

[0076] In the specific calculation, refer to Fig.10 As shown, the intersection line L 2 It can be calculated by the following formula (2):

[0077] L 2 =PQT-QPT formula (2)

[0078] Among them, P represents the target plane, Q represents the preset plane, and L 2 Represents the intersection line of the target plane P and the preset plane Q in three-dimensional space. 2 After that, we can cross-product the antisymmetric matrix [l 2 ]x represents the intersection line L 2 The image l on the image plane 2 , the specific calculation is as follows formula (3):

[0079] [l 2]x=PL 2 PT formula (3)

[0080] Among them, [l 2 ]x represents vector l 2 The cross product antisymmetric matrix, P represents the target plane, L 2 Represents the intersection line between the target plane P and the preset plane Q in three-dimensional space.

[0081] In step S920, when the position identification line and the frame of the image to be processed have an intersection on the image plane, a correction region is determined in the image to be processed according to the plane parameters, and correction is performed on the correction region according to the correction matrix to obtain a corrected image.

[0082] Since the image to be processed is obtained through a photosensitive element, the size of the image to be processed on the image plane is limited by the photosensitive element. However, the position identification line obtained by plane parameter calculation is not limited by the photosensitive element. Therefore, the position of the position identification line on the image plane may intersect with the image to be processed, or it may be completely outside the image to be processed. For example, Fig.10 and Fig.11 As shown in 2 There may be an intersection with the image to be processed, or there may be no intersection with the image to be processed.

[0083] In an exemplary embodiment, referring to Fig.11 As shown in FIG. 1 , when the position identification line and the frame of the image to be processed have an intersection on the image plane, it can be indicated that there is a part in the image to be processed that is not suitable for correction by the correction matrix corresponding to the target plane. At this time, the correction area can be determined in the image to be processed according to the plane parameters to avoid the part that is not suitable for correction by the correction matrix corresponding to the target plane from affecting the correction result.

[0084] The frame refers to the outermost edge line of the image to be processed. In an infinitely large image plane, the data collected by the photosensitive element can be imaged in a certain area of ​​the image plane to form an image to be processed. At this time, the boundary line of the area is the frame of the image to be processed.

[0085] In an exemplary embodiment, before determining whether the position identification line and the frame of the image to be processed have an intersection on the image plane, the frame of the image to be processed may be enhanced to obtain an enhanced frame. Specifically, the frame of the image to be processed may be enhanced by a preset enhancement rule to increase the area of ​​the region enclosed by the frame on the image plane. By enhancing the frame, the area of ​​the region enclosed by the frame on the image plane is increased, and it may be more robustly determined whether the frame and the position identification line have an intersection.

[0086] In an exemplary embodiment, the preset enhancement rule may include a preset value, and the border of the image to be processed is enlarged according to the preset value to form an enhanced border. Fig.12 As shown, preset values ​​e and f can be set to translate and extend the original border 1210 to form an enhanced border 1220, so that the area it encloses increases from W*H to (W+2e)*(H+2f).

[0087] In another exemplary embodiment, the preset enhancement rule may further include calculating the circumscribed ellipse of the original border 1310, and then using the circumscribed ellipse as the enhanced border 1320. Fig.13 shown.

[0088] It should be noted that, through the above enhancement processing, the robustness of determining whether there is an intersection between the position identification line and the frame can be improved. In addition, other enhancement processing methods can also be used to enhance the frame, and the present disclosure does not make special restrictions on this.

[0089] In an exemplary embodiment, when it is determined that the position identification line and the frame of the image to be processed have an intersection on the image plane, a correction area where no correction error will occur can be determined in the image to be processed according to the plane parameters.

[0090] In an exemplary embodiment, the plane point cloud data corresponding to the target plane can be directly determined in the point cloud data corresponding to the image to be processed according to the plane parameters, and then the correction area can be determined in the image to be processed according to the plane point cloud data.

[0091] Optionally, an imaging area 1420 of the plane point cloud data corresponding to the target plane on the image to be processed 1410 may be calculated, and the imaging area 1420 may be determined as the correction area, such as Fig.14 As shown in the figure, since the point cloud data of the target plane can accurately determine the area in the image to be processed when the target plane is imaged, directly using the imaging area of ​​the plane point cloud data on the image to be processed as the correction area can avoid the points outside the vanishing line range, thereby avoiding the problem of distortion in the correction.

[0092] Optionally, when the position identification line and the border of the image to be processed intersect on the image plane, the position identification line can divide the image to be processed into near scene imaging that is closer than a specific line or plane distance, and far scene imaging that is farther than the specific line or plane distance based on the principle of perspective transformation, thereby determining the correction area in the image to be processed.

[0093] In an exemplary embodiment, after the image to be processed is divided into a first area and a second area according to the position identification line, a target area can be determined in the first area and the second area according to the plane point cloud data and the position identification line, and then a correction area can be determined based on the target area.

[0094] In an exemplary embodiment, in order to obtain surface information on the target plane, the target plane may be located closer to the camera, so the first area or the second area of ​​the target plane region may be determined based on the projection of the plane point cloud data of the target plane in the image to be processed, and then the first area or the second area may be determined as the target area. Fig.15 The image to be processed 1510 and the position identification line 1520 are shown. Through the plane point cloud data, it can be determined that the left side of the position identification line 1520 is the target area 1530.

[0095] Optionally, after determining the target area, the target area can be directly used as the correction area for correction. By directly correcting the target area, the integrity of the correction area can be guaranteed to the greatest extent while avoiding the occurrence of data points that cause distortion. In addition, other methods can be used to determine the correction area based on the target area. For example, in the scenario of scanning paper documents, paper documents are usually rectangular. Therefore, refer to Fig.16 As shown, an inscribed rectangle may be drawn in the target area 1610 , and the image area to be processed enclosed by the inscribed rectangle with the largest area among all the obtained inscribed rectangles is taken as the correction area 1620 .

[0096] Furthermore, before dividing the image to be processed into the first area and the second area according to the position identification line, the position identification line may be offset to prevent the position identification line from being unable to divide all points that may have transformation errors into the same area. Fig.17 As shown, the positional relationship between the target plane and the position identification line 1710 in the image to be processed 1720 can be determined based on the plane point cloud data, and then the position identification line 1710 is offset by a preset distance to the side where the target plane is located to obtain the offset position identification line 1730. Then, the image to be processed can be divided into regions based on the offset position identification line 1730.

[0097] At the same time, when there is no intersection between the position identification line and the border of the image to be processed on the image plane, it means that there is no point in the image to be processed that may cause correction errors. Therefore, the image to be processed can be directly corrected according to the correction matrix to obtain a corrected image.

[0098] The following reference Fig.18 , an exemplary embodiment of the present disclosure is described in detail.

[0099] Step S1810, collecting point cloud data corresponding to the image to be processed by a camera; the image to be processed includes a target plane;

[0100] Step S1820, performing plane detection on the point cloud data to determine plane parameters corresponding to the target plane;

[0101] Step S1830, calculating the position identification line and correction matrix corresponding to the image to be processed based on the plane parameters;

[0102] Step S1840, determining whether there is an intersection between the position identification line and the image to be processed on the image plane;

[0103] Step S1850, when the position identification line and the image to be processed have an intersection on the image plane, a correction region is determined in the image to be processed according to the plane parameters, and correction is performed on the correction region according to the correction matrix to obtain a corrected image;

[0104] Step S1860: When there is no intersection between the position identification line and the image to be processed on the image plane, the image to be processed is corrected according to the correction matrix to obtain a corrected image.

[0105] In summary, in this exemplary embodiment, the distance in the three-dimensional space can be reflected on the image plane of the camera through the position identification line, and then, based on whether there is an intersection between the position identification line and the image to be processed on the image plane, it is determined whether there are points in the image to be processed that may cause correction errors. When there is an intersection between the position identification line and the image to be processed, it means that there are points in the image to be processed that may cause correction errors. At this time, the correction area can be determined in the image to be processed based on the plane point cloud data, and then the correction area can be corrected according to the correction matrix corresponding to the target plane. In this way, the problem of correction errors caused by the existence of points other than vanishing lines when performing perspective transformation through the correction matrix can be avoided.

[0106] It should be noted that the above figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0107] For further reference, Fig.19 As shown, in the embodiment of this example, a plane correction device 1900 is also provided, including a parameter calculation module 1910 and a plane correction module 1920. Among them:

[0108] The parameter calculation module 1910 can be used to obtain the plane parameters of the target plane in the image to be processed, and calculate the position identification line and correction matrix corresponding to the image to be processed according to the plane parameters; wherein the position identification line is on the image plane of the image to be processed.

[0109] The plane correction module 1920 can be used to determine the correction area in the image to be processed according to the plane parameters when the position identification line and the frame of the image to be processed have an intersection on the image plane, and correct the correction area according to the correction matrix to obtain a corrected image.

[0110] In an exemplary embodiment, the plane correction module 1920 may be used to correct the image to be processed according to the correction matrix to obtain a corrected image when there is no intersection between the position identification line and the border of the image to be processed on the image plane.

[0111] In an exemplary embodiment, the parameter calculation module 1910 can be used to obtain the camera intrinsic parameters corresponding to the camera that captures the image to be processed; based on the camera intrinsic parameters and the plane parameters, calculate the vanishing line formed on the image plane when the target plane extends to infinity, and use the vanishing line as the position identification line.

[0112] In an exemplary embodiment, the parameter calculation module 1910 can be used to determine the intersection line between the target plane and the preset plane based on the plane parameters; wherein the preset plane is parallel to the image plane, and the distance between the preset plane and the image plane is a preset distance; calculate the cross-sectional line formed by the intersection line in the image plane, and use the cross-sectional line as the position identification line.

[0113] In an exemplary embodiment, the plane correction module 1920 can be used to obtain plane point cloud data corresponding to the target plane in the point cloud data corresponding to the image to be processed according to the plane parameters; and determine the correction area in the image to be processed according to the plane point cloud data.

[0114] In an exemplary embodiment, the plane correction module 1920 may be used to calculate an imaging area of ​​the plane point cloud data on the image to be processed, and determine the imaging area as the correction area.

[0115] In an exemplary embodiment, the plane correction module 1920 can be used to divide the image to be processed into a first area and a second area according to the position identification line; determine the target area in the first area and the second area according to the plane point cloud data and the position identification line, and determine the correction area based on the target area.

[0116] In an exemplary embodiment, the plane correction module 1920 can be used to calculate the imaging area of ​​the plane point cloud data on the image to be processed; when the imaging area is located in the first area, the first area is determined as the target area; when the imaging area is located in the second area, the second area is determined as the target area.

[0117] In an exemplary embodiment, the plane correction module 1920 may be used to generate an inscribed rectangle in the target area, and determine the area surrounded by the inscribed rectangle with the largest area among all the inscribed rectangles as the correction area.

[0118] In an exemplary embodiment, the plane correction module 1920 can be used to determine the positional relationship between the target plane and the position identification line in the image to be processed based on the plane point cloud data; based on the positional relationship, the position identification line is offset by a preset distance to the side where the target plane is located to obtain the offset position identification line.

[0119] In an exemplary embodiment, referring to Fig. 20 As shown, the plane correction device 2000 also includes a border enhancement module 2010, which is used to perform image enhancement processing on the border of the image to be processed to obtain an enhanced border of the image to be processed.

[0120] In an exemplary embodiment, the border enhancement module 2010 may be used to enhance the border of the image to be processed according to a preset enhancement rule, so as to increase the area of ​​the region surrounded by the border on the image plane.

[0121] In an exemplary embodiment, the parameter calculation module 1910 can be used to perform plane detection based on the point cloud data of the image to be processed and obtain plane parameters corresponding to the target plane; wherein the target plane is the plane that occupies the largest area of ​​the image to be processed among all the detected planes.

[0122] The specific details of each module in the above device have been described in detail in the implementation method of the method part. The undisclosed details can be found in the implementation method of the method part, so they will not be repeated here.

[0123] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0124] The exemplary embodiments of the present disclosure also provide a computer-readable storage medium on which a program product capable of implementing the above-mentioned method of the present specification is stored. In some possible implementations, various aspects of the present disclosure can also be implemented in the form of a program product, which includes program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary embodiments of the present disclosure described in the above-mentioned "Exemplary Method" section of the present specification, for example, it can execute Fig. 9 and Fig.18 Any one or more steps in .

[0125] It should be noted that the computer-readable medium shown in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0126] In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, apparatus, or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0127] In addition, program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0128] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0129] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A plane correction method, characterized in that: include: Acquire plane parameters of a target plane in the image to be processed, and calculate a position identification line and a correction matrix corresponding to the image to be processed according to the plane parameters; wherein the position identification line is on the image plane of the image to be processed; When the position identification line and the frame of the image to be processed have an intersection on the image plane, a correction region is determined in the image to be processed according to the plane parameters, and correction is performed on the correction region according to the correction matrix to obtain a corrected image, wherein the frame of the image to be processed is an edge line of the outermost circle of the image to be processed, The step of calculating the position identification line corresponding to the image to be processed according to the plane parameters comprises: obtaining the camera intrinsic parameters corresponding to the camera that photographs the image to be processed, and calculating the vanishing line formed on the image plane when the target plane extends to infinity according to the camera intrinsic parameters and the plane parameters, and using the vanishing line as the position identification line; or determining the intersection line of the target plane and a preset plane according to the plane parameters, wherein the preset plane is parallel to the image plane, and the distance between the preset plane and the image plane is a preset distance, calculating the cross-sectional line formed by the intersection line on the image plane, and using the cross-sectional line as the position identification line. Wherein, determining the correction area in the image to be processed according to the plane parameters includes: acquiring plane point cloud data corresponding to the target plane in the point cloud data corresponding to the image to be processed according to the plane parameters; determining the correction area in the image to be processed according to the plane point cloud data, Wherein, determining the correction area in the image to be processed according to the plane point cloud data includes: dividing the image to be processed into a first area and a second area according to the position identification line; determining a target area in the first area and the second area according to the plane point cloud data and the position identification line, and determining the correction area based on the target area, wherein determining the correction area based on the target area includes: generating an inscribed rectangle in the target area, and determining the area enclosed by the inscribed rectangle with the largest area among all the inscribed rectangles as the correction area.

2. The method according to claim 1, characterized in that The method further comprises: When there is no intersection between the position identification line and the frame of the image to be processed on the image plane, the image to be processed is corrected according to the correction matrix to obtain a corrected image.

3. The method according to claim 1, characterized in that Determining a correction area in the image to be processed according to the planar point cloud data also includes: An imaging area of ​​the planar point cloud data on the image to be processed is calculated, and the imaging area is determined as a correction area.

4. The method according to claim 1, characterized in that: The step of determining a target area in the first area and the second area according to the plane point cloud data and the position identification line includes: Calculate the imaging area of ​​the plane point cloud data on the image to be processed; When the imaging area is located in a first area, determining the first area as a target area; When the imaging area is located in the second area, the second area is determined to be the target area.

5. The method according to claim 1, characterized in that: Before dividing the image to be processed into a first area and a second area according to the position identification line, the method further includes: Determine the positional relationship between the target plane and the position identification line in the image to be processed according to the plane point cloud data; The position identification line is offset by a preset distance toward the side where the target plane is located according to the position relationship to obtain the offset position identification line.

6. The method according to claim 1, characterized in that The method further comprises: Performing image enhancement processing on the border of the image to be processed to obtain an enhanced border of the image to be processed.

7. The method according to claim 6, characterized in that The performing image enhancement processing on the border of the image to be processed includes: The border of the image to be processed is enhanced according to a preset enhancement rule to increase the area of ​​the region surrounded by the border on the image plane.

8. The method according to claim 1, characterized in that Get the plane parameters of the target plane in the image to be processed, including: Plane detection is performed based on the point cloud data of the image to be processed, and plane parameters corresponding to the target plane are obtained; wherein the target plane is the plane that occupies the largest area of ​​the image to be processed among all the detected planes.

9. A plane correction device, characterized in that: include: A parameter calculation module, used to obtain the plane parameters of the target plane in the image to be processed, and calculate the position identification line and the correction matrix corresponding to the image to be processed according to the plane parameters; wherein the position identification line is on the image plane of the image to be processed; A plane correction module is used to determine a correction area in the image to be processed according to the plane parameters when there is an intersection between the position identification line and the frame of the image to be processed on the image plane, and to perform correction on the correction area according to the correction matrix to obtain a corrected image, wherein the frame of the image to be processed refers to the edge line of the outermost circle of the image to be processed, The step of calculating the position identification line corresponding to the image to be processed according to the plane parameters comprises: obtaining the camera intrinsic parameters corresponding to the camera that photographs the image to be processed, and calculating the vanishing line formed on the image plane when the target plane extends to infinity according to the camera intrinsic parameters and the plane parameters, and using the vanishing line as the position identification line; or determining the intersection line of the target plane and a preset plane according to the plane parameters, wherein the preset plane is parallel to the image plane, and the distance between the preset plane and the image plane is a preset distance, calculating the cross-sectional line formed by the intersection line on the image plane, and using the cross-sectional line as the position identification line. Wherein, determining the correction area in the image to be processed according to the plane parameters includes: acquiring plane point cloud data corresponding to the target plane in the point cloud data corresponding to the image to be processed according to the plane parameters; determining the correction area in the image to be processed according to the plane point cloud data, Wherein, determining the correction area in the image to be processed according to the plane point cloud data includes: dividing the image to be processed into a first area and a second area according to the position identification line; determining a target area in the first area and the second area according to the plane point cloud data and the position identification line, and determining the correction area based on the target area, wherein determining the correction area based on the target area includes: generating an inscribed rectangle in the target area, and determining the area enclosed by the inscribed rectangle with the largest area among all the inscribed rectangles as the correction area.

10. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 8 by executing the executable instructions.

Images