Image distortion correction method, device, driving device, and medium

Abstract

The application provides a kind of image distortion correction method, equipment, driving equipment and medium, wherein method is applied in GPU, the method includes extracting the image coordinates of the image to be corrected, and determining the distortion mapping data of the camera terminal according to the camera terminal parameters;Based on the image coordinates of the image to be corrected and the distortion mapping data of the camera terminal, the distorted image is corrected to obtain the corrected image, so that the distortion mapping data of the camera terminal can be calculated in real time based on the strong computing power of GPU, and after extracting the coordinates of the image to be corrected, the distorted image is corrected, so that the process of calculating the distortion mapping data and the image correction are integrated. Compared with using CPU to calculate the distortion mapping data of the camera terminal and transmitting the distortion mapping data of the camera terminal between GPU, the time consumed is relatively shorter, so as to improve the efficiency of image distortion correction.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and specifically provides an image distortion correction method, device, driving device, and medium. Background Technology

[0002] Image distortion caused by inherent defects in imaging devices such as cameras is difficult to avoid. Therefore, distortion correction of images acquired using imaging devices is a basic image processing operation. Distortion correction can ensure that computer vision systems equipped with imaging devices can work normally and accurately.

[0003] Typically, the CPU first calculates the mapping relationship between the original image and the distorted image as the distortion mapping data for the imaging device. Then, the GPU copies the distortion mapping data from the CPU and generates the distorted image based on the distortion mapping data.

[0004] However, images captured by camera terminals are updated in real time. In real-world scenarios, each frame requires distortion correction. If the camera terminal's image preprocessing is slow, it will severely impact the efficiency and accuracy of subsequent processes, such as real-time issues related to vehicle lane detection, obstacle recognition, and vehicle control. Furthermore, the CPU is slow in calculating distortion mapping data, and the resulting data is large. Sending this mapping relationship to the GPU is also relatively slow, leading to low efficiency in image distortion correction. Summary of the Invention

[0005] To overcome the above-mentioned shortcomings, this invention is proposed to provide an image distortion correction method, apparatus, driving device, and medium that solves, or at least partially solves, the technical problem that the CPU is slow in calculating distorted mapping data, the calculated distorted mapping data is large in volume, and the transmission speed when sending the mapping relationship to the GPU is also relatively slow, resulting in low efficiency of image distortion correction.

[0006] In a first aspect, the present invention provides an image distortion correction method, wherein the image distortion correction method is applied in a GPU, the method comprising:

[0007] Extract the image coordinates of the image to be corrected, and determine the distortion mapping data of the camera terminal based on the camera terminal parameters;

[0008] Based on the image coordinates of the image to be corrected and the distortion mapping data of the camera terminal, the distorted image is corrected to obtain a corrected image.

[0009] Furthermore, in the image distortion correction method described above, extracting the image coordinates of the image to be corrected and determining the distortion mapping data of the camera terminal based on the camera terminal parameters includes:

[0010] If the required value of the camera terminal in the current scene is detected to be greater than or equal to a preset threshold, the image coordinates of the image to be corrected are extracted, and the distortion mapping data of the camera terminal is determined according to the camera terminal parameters.

[0011] Furthermore, the image distortion correction method described above also includes:

[0012] If the required value of the camera terminal in the current scene is less than the preset threshold, the distortion mapping data of the camera terminal is obtained from the CPU and / or memory. After extracting the image coordinates of the image to be corrected, the distorted image is corrected based on the image coordinates of the image to be corrected and the distortion mapping data of the camera terminal to obtain a corrected image.

[0013] Furthermore, in the image distortion correction method described above, the process of obtaining the camera terminal parameters includes:

[0014] Obtain the calibration image of the calibration plate;

[0015] Based on the calibration image, determine the world coordinates of each corner point in the calibration plate in the world coordinate system and the image coordinates in the calibration image;

[0016] Based on the world coordinates and the image coordinates, the camera terminal is calibrated to obtain the camera terminal parameters.

[0017] Furthermore, in the image distortion correction method described above, obtaining the calibration image of the calibration plate includes:

[0018] A preset number of images of the calibration board are captured within a preset range area, wherein each captured image is configured to completely cover the preset range area after being superimposed.

[0019] Furthermore, in the image distortion correction method described above, the distorted image is corrected based on the image coordinates of the image to be corrected and the distortion mapping data of the camera terminal to obtain a corrected image, including:

[0020] Based on the image coordinates of the image to be corrected and the distortion mapping data of the camera terminal, the distorted image is corrected by an interpolation algorithm to obtain a corrected image.

[0021] Furthermore, in the image distortion correction method described above, the camera terminal parameters include camera terminal intrinsic parameters, camera terminal extrinsic parameters, and camera terminal distortion parameters.

[0022] In a second aspect, the present invention provides an image distortion correction device, the image distortion correction device comprising a GPU and a storage device, the storage device being adapted to store a plurality of program codes, the program codes being adapted to be loaded and run by the GPU to perform the image distortion correction method described above.

[0023] In a third aspect, a driving device is provided, the driving device including the image distortion correction device as described above.

[0024] In a third aspect, a computer-readable storage medium is provided, the computer-readable storage medium storing a plurality of program codes adapted to be loaded and executed by a processor to perform the image distortion correction method described in any of the preceding claims.

[0025] The above-described technical solutions of the present invention have at least one or more of the following beneficial effects:

[0026] In implementing the technical solution of this invention, by extracting the image coordinates of the image to be corrected and determining the distortion mapping data of the camera terminal according to the camera terminal parameters, the distorted image is corrected based on the image coordinates of the image to be corrected and the distortion mapping data of the camera terminal to obtain a corrected image. This allows for real-time calculation of the distortion mapping data of the camera terminal based on the strong computing power of the GPU, and after extracting the coordinates of the image to be corrected, distortion correction is performed on the image to be corrected, thus integrating the calculation of distortion mapping data with the image correction process. Compared to using the CPU to calculate the distortion mapping data of the camera terminal and transmitting the distortion mapping data between the CPU and the GPU, the time consumed is relatively short, thereby improving the efficiency of image distortion correction. Attached Figure Description

[0027] The disclosure of this invention will become more readily understood with reference to the accompanying drawings. It will be readily understood by those skilled in the art that these drawings are for illustrative purposes only and are not intended to limit the scope of protection of this invention. Furthermore, similar numbers in the drawings are used to denote similar components, wherein:

[0028] Figure 1 This is a schematic flowchart of the main steps of an image distortion correction method according to an embodiment of the present invention;

[0029] Figure 2 This is a flowchart illustrating the process of obtaining camera terminal parameters;

[0030] Figure 3 This is a schematic flowchart of the main steps of an image distortion correction method according to an embodiment of the present invention;

[0031] Figure 4This is a main structural block diagram of an image distortion correction device according to an embodiment of the present invention. Detailed Implementation

[0032] Some embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0033] In the description of this invention, "module" and "processor" can include hardware, software, or a combination of both. A module can include hardware circuitry, various suitable sensors, communication ports, memory, and may also include software components, such as program code, or a combination of software and hardware. A processor can be a central processing unit, microprocessor, image processor, digital signal processor, or any other suitable processor. The processor has data and / or signal processing capabilities. The processor can be implemented in software, in hardware, or a combination of both. Non-transitory computer-readable storage media includes any suitable medium capable of storing program code, such as magnetic disks, hard disks, optical disks, flash memory, read-only memory, random access memory, etc. The term "A and / or B" means all possible combinations of A and B, such as only A, only B, or A and B. The terms "at least one A or B" or "at least one of A and B" have a similar meaning to "A and / or B" and can include only A, only B, or A and B. The singular terms "a" or "this" can also include plural forms.

[0034] Typically, when performing image base table correction, the CPU first calculates the mapping relationship between the original image and the distorted image as the distortion mapping data of the imaging device. Then, the GPU copies the distortion mapping data from the CPU and generates the distorted image based on the distortion mapping data.

[0035] However, images captured by camera terminals are updated in real time. In real-world scenarios, each frame requires distortion correction. If the camera terminal's image preprocessing is slow, it will severely impact the efficiency and accuracy of subsequent processes, such as real-time issues related to vehicle lane detection, obstacle recognition, and vehicle control. Furthermore, the CPU is slow in calculating distortion mapping data, and the resulting data is large. Sending this mapping relationship to the GPU is also relatively slow, leading to low efficiency in image distortion correction.

[0036] Therefore, in order to solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0037] See appendix Figure 1 , Figure 1This is a schematic flowchart illustrating the main steps of an image distortion correction method according to an embodiment of the present invention. Figure 1 As shown, the image distortion correction method in this embodiment of the invention mainly includes the following steps 101-102.

[0038] Step 101: Extract the image coordinates of the image to be corrected, and determine the distortion mapping data of the camera terminal based on the camera terminal parameters;

[0039] In one specific implementation, a camera terminal can capture images in real time, obtaining a distorted image of the current environment as the image to be corrected. After acquiring the image to be corrected, the GPU can extract its image coordinates and determine the distortion mapping data of the camera terminal based on the camera terminal parameters. This distortion mapping data can include the mapping relationship between the real image and the distorted image. The real image is the image after distortion correction of the distorted image, i.e., the subsequently corrected image.

[0040] In a specific implementation, the process of acquiring the camera terminal parameters can be referred to Figure 2 As shown, Figure 2 This is a flowchart illustrating the process of obtaining camera terminal parameters, such as... Figure 2 As shown, the process of obtaining camera terminal parameters in this embodiment may include the following steps 201-203:

[0041] Step 201: Obtain the calibration image of the calibration board;

[0042] Specifically, a predetermined number of images of the calibration board can be captured within a predetermined range, wherein each captured image is configured to completely cover the predetermined range after being superimposed. The calibration board is a geometric model used in applications such as machine vision, image measurement, photogrammetry, and 3D reconstruction to determine lens distortion and the conversion relationship between physical dimensions and pixels. For example, a checkerboard calibration board can be used. The calibration images can be images captured by a camera terminal with the aforementioned calibration board as the subject.

[0043] Step 202: Based on the calibration image, determine the world coordinates of each corner point in the calibration plate in the world coordinate system and the image coordinates in the calibration image;

[0044] In a specific implementation process, after acquiring the calibration image, calibration methods such as Zhang's calibration method can be used to calibrate the camera terminal and obtain its parameters. In practice, in image measurement and machine vision applications, to determine the relationship between the three-dimensional geometric position of a point on the surface of a spatial object and its corresponding point in the image, it is necessary to determine the camera parameters. The process of solving for these parameters is called camera terminal calibration.

[0045] Specifically, taking a checkerboard calibration board as an example, the corner points are the vertices of each square in the checkerboard calibration board. The origin of the world coordinate system can be set as the top-left corner point of the checkerboard calibration board, and the length and width of the checkerboard calibration board can be used as the X-axis and Y-axis of the world coordinate system, respectively. The Z-axis of the world coordinate system can be perpendicular to the checkerboard plane. In this way, the world coordinates of each corner point in the calibration board can be obtained. The image coordinates of each corner point in the calibration image can be obtained using functions in a vision software library (such as the OpenCV library).

[0046] Step 203: Based on the world coordinates and the image coordinates, calibrate the camera terminal to obtain the camera terminal parameters.

[0047] In a specific implementation process, the camera terminal can be calibrated based on the world coordinates and the image coordinates using methods such as Zhang's calibration to obtain the camera terminal parameters. The specific process can be referred to existing related technologies and will not be elaborated here.

[0048] In a specific implementation, after obtaining the camera terminal parameters, the distortion mapping data of the camera terminal can be determined based on these parameters. For the specific implementation process, please refer to existing related technologies, which will not be elaborated here. This distortion mapping data can include, but is not limited to, the mapping relationship between the coordinates of the real image and the coordinates of the distorted image.

[0049] In a specific implementation, since the GPU has higher computing power than the CPU, the process of determining the distortion mapping data of the camera terminal is more efficient than reading from the CPU or from memory. This can shorten the entire image correction process and improve image correction efficiency.

[0050] Step 102: Based on the image coordinates of the image to be corrected and the distortion mapping data of the camera terminal, the distorted image is corrected to obtain a corrected image.

[0051] In a specific implementation, the distorted image can be corrected using an interpolation algorithm based on the image coordinates of the image to be corrected and the distortion mapping data of the camera terminal, thus obtaining a corrected image. The interpolation algorithm can be, but is not limited to, bilinear interpolation. Specifically, the correction process can refer to existing related technologies, which will not be elaborated here.

[0052] The image distortion correction method in this embodiment extracts the image coordinates of the image to be corrected and determines the distortion mapping data of the camera terminal based on the camera terminal parameters. Then, based on the image coordinates of the image to be corrected and the distortion mapping data of the camera terminal, the distorted image is corrected to obtain a corrected image. In this way, the distortion mapping data of the camera terminal can be calculated in real time based on the strong computing power of the GPU. After extracting the coordinates of the image to be corrected, distortion correction is performed on the image to be corrected. This integrates the process of calculating distortion mapping data and image correction. Compared with using the CPU to calculate the distortion mapping data of the camera terminal and transmitting the distortion mapping data between the CPU and the GPU, the time consumed is relatively short, thereby improving the efficiency of image distortion correction.

[0053] In a specific implementation process, although the GPU has high computing power, the GPU's real-time calculation of the distortion mapping data of the camera terminal will also affect the GPU's intrinsic parameters to a certain extent. Therefore, the present invention provides the following embodiments.

[0054] See appendix Figure 3 , Figure 3 This is a schematic flowchart illustrating the main steps of an image distortion correction method according to an embodiment of the present invention. Figure 3 As shown, the image distortion correction method in this embodiment of the invention mainly includes the following steps 301-304.

[0055] Step 301: Detect whether the demand value for using the camera terminal in the current scene is less than a preset threshold; if yes, proceed to step 302; if no, proceed to step 303.

[0056] Step 302: Obtain the distortion mapping data of the camera terminal from the CPU and / or memory;

[0057] Step 303: Extract the image coordinates of the image to be corrected, and determine the distortion mapping data of the camera terminal based on the camera terminal parameters;

[0058] Step 304: Based on the image coordinates of the image to be corrected and the distortion mapping data of the camera terminal, correct the distorted image to obtain a corrected image.

[0059] In a specific implementation, the demand value of the camera terminal in the current scene can be detected in real time, and it can be detected whether the demand value of the camera terminal in the current scene is less than a preset threshold. If the demand value of the camera terminal in the current scene is detected to be greater than or equal to the preset threshold, the image coordinates of the image to be corrected are extracted, and the distortion mapping data of the camera terminal is determined according to the camera terminal parameters. If the demand value of the camera terminal in the current scene is detected to be less than the preset threshold, the distortion mapping data of the camera terminal is obtained from the CPU and / or memory. After extracting the image coordinates of the image to be corrected, the distorted image is corrected based on the image coordinates of the image to be corrected and the distortion mapping data of the camera terminal to obtain a corrected image.

[0060] Specifically, for vehicles traveling at high speeds, the real-time demand for images is relatively high, resulting in a higher demand value for the camera terminal in the current scene. In this case, the GPU is used to calculate the distortion mapping data of the camera terminal. However, when the vehicle is stopped or entering a garage at low speeds, the real-time demand for images is relatively low, resulting in a lower demand value for the camera terminal in the current scene. In this case, the distortion mapping data of the camera terminal can be obtained from the CPU and / or memory, thereby freeing up the GPU's operating load.

[0061] It should be noted that although the steps in the above embodiments are described in a specific order, those skilled in the art will understand that in order to achieve the effects of the present invention, different steps do not necessarily have to be executed in such an order. They can be executed simultaneously (in parallel) or in other orders, and these variations are all within the scope of protection of the present invention.

[0062] Those skilled in the art will understand that all or part of the processes in the method of the above embodiment of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable storage medium can include any entity or device capable of carrying the computer program code, a medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory, a random access memory, an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc. It should be noted that the content included in the computer-readable storage medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable storage medium does not include electrical carrier signals and telecommunication signals.

[0063] Furthermore, the present invention also provides an image distortion correction device.

[0064] See appendix Figure 4 , Figure 4 This is a main structural block diagram of an image distortion correction device according to an embodiment of the present invention. Figure 4 As shown, the error compensation device of the multi-level power grid dispatching system in this embodiment of the invention may include a processor 41 and a storage device 42.

[0065] The storage device 42 can be configured to store a program for executing the error compensation method of the multi-level grid dispatching system described in the above-described method embodiments. The processor 41 can be configured to execute the program in the storage device 42, which includes, but is not limited to, a program for executing the error compensation method of the multi-level grid dispatching system described in the above-described method embodiments. For ease of explanation, only the parts related to the embodiments of the present invention are shown; for specific technical details not disclosed, please refer to the method section of the embodiments of the present invention. The error compensation device of the multi-level grid dispatching system can be a control device comprising various electronic devices.

[0066] In a specific implementation, there can be multiple storage devices 42 and processors 41. The program executing the error compensation method of the multi-level grid dispatching system described in the above method embodiment can be divided into multiple subroutines. Each subroutine can be loaded and run by the processor 41 to execute different steps of the error compensation method of the multi-level grid dispatching system described in the above method embodiment. Specifically, each subroutine can be stored in different storage devices 42, and each processor 41 can be configured to execute programs in one or more storage devices 42 to jointly implement the error compensation method of the multi-level grid dispatching system described in the above method embodiment. That is, each processor 41 executes different steps of the error compensation method of the multi-level grid dispatching system described in the above method embodiment to jointly implement the error compensation method of the multi-level grid dispatching system described in the above method embodiment.

[0067] The aforementioned multiple processors 41 can be processors deployed on the same device. For example, the device can be a high-performance device composed of multiple processors, and the multiple processors 41 can be processors configured on that high-performance device. Alternatively, the aforementioned multiple processors 41 can also be processors deployed on different devices. For example, the device can be a server cluster, and the multiple processors 41 can be processors on different servers within the server cluster.

[0068] Furthermore, the present invention also provides a driving device. This driving device may include the image distortion correction device described in the above embodiments.

[0069] Furthermore, the present invention also provides a computer-readable storage medium. In one embodiment of the computer-readable storage medium according to the present invention, the computer-readable storage medium can be configured to store a program for performing the image distortion correction method of the above-described method embodiments. This program can be loaded and run by a processor to implement the above-described image distortion correction method. For ease of explanation, only the parts related to the embodiments of the present invention are shown; for specific technical details not disclosed, please refer to the method section of the embodiments of the present invention. The computer-readable storage medium can be a storage device comprising various electronic devices. Optionally, in the embodiments of the present invention, the computer-readable storage medium is a non-transitory computer-readable storage medium.

[0070] Furthermore, it should be understood that since the various modules are only provided to illustrate the functional units of the device of the present invention, the physical devices corresponding to these modules may be the processor itself, or a part of the processor's software, hardware, or a combination of software and hardware. Therefore, the number of modules shown in the figures is merely illustrative.

[0071] Those skilled in the art will understand that the various modules in the device can be adaptively split or combined. Such splitting or combining of specific modules will not cause the technical solution to deviate from the principles of the present invention; therefore, the technical solutions after splitting or combining will fall within the protection scope of the present invention.

[0072] It should be noted that all vehicle-related operations mentioned in this invention, such as acquiring image frames through the vehicle's visual sensors and performing autonomous driving control, are executed only after full authorization from the user or other parties. In other words, the vehicle in this invention is an authorized vehicle. In some embodiments, the vehicle's infotainment system or a backend server can detect whether authorization information has been received. If authorization information is received, it indicates that the current vehicle is authorized; otherwise, the current vehicle is unauthorized. The authorization information can be sent through terminal devices including, but not limited to, mobile phones, tablets, and smartwatches.

[0073] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after such changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. An image distortion correction method, characterized in that, When applied to a GPU, the method includes: Extract the image coordinates of the image to be corrected, and determine the distortion mapping data of the camera terminal based on the camera terminal parameters; Based on the image coordinates of the image to be corrected and the distortion mapping data of the camera terminal, the distorted image is corrected to obtain the corrected image; Extracting the image coordinates of the image to be corrected, and determining the distortion mapping data of the camera terminal based on the camera terminal parameters, including: If the required value of the camera terminal in the current scene is detected to be greater than or equal to a preset threshold, the image coordinates of the image to be corrected are extracted, and the distortion mapping data of the camera terminal is determined according to the camera terminal parameters. If the required value of the camera terminal in the current scene is detected to be less than the preset threshold, the distortion mapping data of the camera terminal is obtained from the CPU and / or memory so that after extracting the image coordinates of the image to be corrected, the distorted image is corrected based on the image coordinates of the image to be corrected and the distortion mapping data of the camera terminal to obtain a corrected image. The demand value represents the real-time demand for images in the current scene.

2. The image distortion correction method according to claim 1, characterized in that, The process of acquiring the camera terminal parameters includes: Obtain the calibration image of the calibration plate; Based on the calibration image, determine the world coordinates of each corner point in the calibration plate in the world coordinate system and the image coordinates in the calibration image; Based on the world coordinates and the image coordinates, the camera terminal is calibrated to obtain the camera terminal parameters.

3. The image distortion correction method according to claim 2, characterized in that, Obtain the calibration image of the calibration board, including: A preset number of images of the calibration board are captured within a preset range area, wherein each captured image is configured to completely cover the preset range area after being superimposed.

4. The image distortion correction method according to claim 1, characterized in that, Based on the image coordinates of the image to be corrected and the distortion mapping data of the camera terminal, the distorted image is corrected to obtain a corrected image, including: Based on the image coordinates of the image to be corrected and the distortion mapping data of the camera terminal, the distorted image is corrected by an interpolation algorithm to obtain a corrected image.

5. The image distortion correction method according to claim 1, characterized in that, The camera terminal parameters include camera terminal internal parameters, camera terminal external parameters, and camera terminal distortion parameters.

6. An image distortion correction device, characterized in that, The method includes a GPU and a storage device, the storage device being adapted to store a plurality of program codes, the program codes being adapted to be loaded and run by the GPU to perform the image distortion correction method according to any one of claims 1 to 5.

7. A driving device, characterized in that, Includes the image distortion correction device as described in claim 6.

8. A computer-readable storage medium, characterized in that, The system stores multiple lines of program code, which are adapted to be loaded and run by a processor to perform the image distortion correction method according to any one of claims 1 to 5.

Images