Fisheye camera image correction method

Abstract

The invention provides a fisheye camera image correction method, relates to the technical field of image processing, and solves the technical problem that fisheye camera correction needs to depend on a specific calibration template and manual intervention in an image correction process, resulting in high implementation difficulty. The method comprises the following steps: acquiring a distorted image to be corrected, and determining pixel coordinates of each pixel in the distorted image; mapping the pixel coordinates into a three-dimensional coordinate system to obtain three-dimensional coordinates of each pixel in the three-dimensional coordinate system; projecting each three-dimensional coordinate into a preset correction image, and determining a correction coordinate corresponding to each pixel in the correction image; and mapping the pixel value of each pixel to the corresponding correction coordinate to obtain a corrected image. The method is used in the image correction process of the fisheye camera.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a method for correcting fisheye camera images. Background Art

[0002] Fisheye cameras, with their uniquely wide field of view, are widely used in security surveillance, autonomous driving, panoramic imaging, and other fields. The images captured by fisheye cameras have an extremely large field of view, allowing them to cover a very large monitoring area. However, this wide field of view can cause distortion in the captured images, with the shapes of objects in the fisheye image significantly deviating from their actual shapes.

[0003] Currently, mainstream methods for fisheye image correction include traditional algorithms based on physical calibration plates and intelligent algorithms based on deep learning. Methods based on grid-like or checkerboard calibration plates require manual operation of the camera to capture multi-view calibration images to calculate correction parameters. These methods are difficult to implement when the camera is in a closed environment, on an overhead platform, or inaccessible to direct view. These methods rely on specific calibration templates and manual intervention. Summary of the Invention

[0004] The present application provides a fisheye camera image correction method, which solves the technical problem that fisheye camera correction requires reliance on specific calibration templates and manual intervention, resulting in greater difficulty in implementation.

[0005] To achieve the above objectives, this application adopts the following technical solutions: In a first aspect, a fisheye camera image correction method is provided, comprising: obtaining a distorted image to be corrected and determining the pixel coordinates of each pixel in the distorted image; mapping the pixel coordinates into a three-dimensional coordinate system to obtain the three-dimensional coordinates of each pixel in the three-dimensional coordinate system; projecting each of the three-dimensional coordinates into a preset corrected image to determine the corrected coordinates corresponding to each pixel in the corrected image; and mapping the pixel value of each pixel into the corresponding corrected coordinates to obtain a corrected image.

[0006] In combination with the first aspect above, in a possible implementation, the process of mapping pixel coordinates into a three-dimensional coordinate system includes: obtaining pixel coordinates of the distorted image; and the center pixel coordinates ;calculate arrive Euclidean distance : ; Determine the virtual angle of incidence ; According to the virtual angle of incidence Calculate the virtual three-dimensional coordinates P(x, y, z) corresponding to the pixel coordinates, and the virtual three-dimensional coordinates satisfy the following formula: .

[0007] In combination with the first aspect above, in a possible implementation, the virtual incident angle , satisfying the following formula:

[0008] in, is the focal length of the fisheye camera, are the optimized polynomial coefficients.

[0009] In combination with the first aspect above, in a possible implementation, the process of mapping the three-dimensional coordinates to the corrected image coordinates includes: , project the virtual three-dimensional coordinates P(x, y, z) into the preset corrected image, and output the pixel coordinates in the corrected image . Among them, the pixel coordinates of any pixel position in the preset correction image , satisfying the following formula:

[0010] in, represents the matrix multiplication operation, is the transposed matrix of the virtual three-dimensional coordinate P(x, y, z); is the virtual three-dimensional coordinate The intermediate homogeneous coordinates after matrix transformation and unnormalization.

[0011] In combination with the first aspect above, in one possible implementation, determining the correction coordinates corresponding to each pixel in the corrected image includes: projecting each three-dimensional coordinate into a preset corrected image to obtain the initial correction coordinates corresponding to each pixel in the corrected image; acquiring the physical straight line target in the distorted image; and determining a projection loss value based on the pixel coordinate points of the physical straight line target in the distorted image and the pixel coordinate points of the physical straight line target in the corrected image; and adjusting the initial correction coordinates corresponding to each pixel in the corrected image based on the projection loss value to determine the correction coordinates corresponding to each pixel in the corrected image.

[0012] In conjunction with the first aspect described above, in one possible implementation, the projection loss value includes a first loss value. Determining the projection loss value based on pixel coordinates of a physical linear target in a distorted image and pixel coordinates of the physical linear target in a corrected image includes: obtaining any three pixel coordinates on the same linear target; determining three first corrected pixel coordinates corresponding to the three pixel coordinates in the corrected image; and determining a first loss value based on the three first corrected pixel coordinates, the first loss value being used to represent the deviation between a line connecting the three first corrected pixel coordinates and the linear line.

[0013] In conjunction with the first aspect described above, in one possible implementation, the projection loss value further includes a second loss value. Determining the projection loss value based on the pixel coordinates of the physical straight line target in the distorted image and the pixel coordinates of the physical straight line target in the corrected image includes: obtaining parallel physical straight line targets in the distorted image and obtaining at least two coordinate points of each parallel physical straight line target. Determining second corrected pixel coordinates corresponding to the two coordinate points of each parallel physical straight line target in the corrected image. Calculating a second loss value for the parallel physical straight line target based on the second corrected pixel coordinates, the second loss value being used to represent a parallel deviation value after projecting the parallel physical straight line target onto a preset corrected image.

[0014] In combination with the first aspect above, in a possible implementation, the first loss Satisfies the following formula:

[0015] Wherein, N represents the number of entity straight line targets in the distorted image, and n represents the number of entity straight line targets among N entity straight line targets. A solid straight line target; The i-th pixel coordinate of a solid straight line target in the distorted image is marked as U n,i ( u n,i , v n,i ), the rectified pixel obtained by mapping the i-th pixel coordinate to the preset rectified image is recorded as U t n,i ( u t n,i ,v t n,i ); are the indices of three different pixel points in the same physical linear target, used to traverse different physical linear targets; C is a set of elements consisting of three different coordinates randomly selected from the pixel coordinates of each linear target in the rectified image, ||C|| represents the number of elements in set C, || ||2 means The L2 norm of .

[0016] In combination with the first aspect above, in a possible implementation, the second loss Satisfy the following calculation formula:

[0017] Among them, Q represents the total number of categories of straight line targets in the distorted image classified according to the actual parallel relationship, Indicates the first nThe parallel solid straight line target of the group; n The coordinates of any two pixels in the parallel solid straight line target are marked as U n,i1 (u n,i1 ,v n,i1 ), U n,i2 (u n,i2 ,v n,i2 ), the second corrected pixel coordinates obtained by mapping the two pixel coordinates to the preset corrected image are marked as U t n,i1 (u t n,i1 ,v t n,i1 ), U t n,i2 (u t n,i2 ,v t n,i2 ); The indexes of two pixels in the nth group of parallel solid line targets in the same parallel category; For The indexes of two pixels on another line in the same set of parallel solid line targets; It means that from each linear target in each category, any two point coordinates are taken from the rectified image pixels to form a set of several point pairs, and the point pairs are combined into a set.

[0018] In combination with the first aspect above, in a possible implementation, the adjustment method includes: iteratively optimizing using a gradient optimization method , Satisfies the following formula:

[0019] in, ; for .

[0020] In a second aspect, an electronic device is provided, comprising a communication unit and a processing unit; the communication unit is used to obtain a distorted image to be corrected; the processing unit is used to determine the pixel coordinates of each pixel in the distorted image; the pixel coordinates are mapped into a three-dimensional coordinate system to obtain the three-dimensional coordinates of each pixel in the three-dimensional coordinate system; each of the three-dimensional coordinates is projected into a preset corrected image to determine the corrected coordinates corresponding to each pixel in the corrected image; and the pixel value of each pixel is mapped into the corresponding corrected coordinates to obtain a corrected image.

[0021] In a third aspect, the present application provides an electronic device comprising: a processor and a storage medium; the storage medium comprises instructions, and the processor is used to execute the instructions to implement the method described in the first aspect and any possible implementation manner of the first aspect.

[0022] In a fourth aspect, the present application provides a computer-readable storage medium, which stores instructions. When the instructions are executed on an electronic device, the electronic device executes the method described in the first aspect and any possible implementation of the first aspect.

[0023] In a fifth aspect, the present application provides a computer program product comprising instructions, which, when executed on an electronic device, enables the electronic device to execute the method as described in the first aspect and any possible implementation manner of the first aspect.

[0024] This application provides a method for correcting fisheye camera images. This method maps the pixel values ​​of the distorted image to the corrected image by mapping the pixel points of the distorted image to a three-dimensional coordinate system, and then mapping the coordinate points in the three-dimensional coordinate system to the corrected image, thereby obtaining a corrected image. Based on this, the fisheye camera image correction method provided in this application can correct distorted images captured by a fisheye camera without the need for a specific calibration template or manual intervention.

[0025] It should be understood that the description of technical features, technical solutions, beneficial effects or similar language in this application does not imply that all features and advantages can be realized in any single embodiment. On the contrary, it is understood that the description of a feature or beneficial effect means that a specific technical feature, technical solution or beneficial effect is included in at least one embodiment. Therefore, the description of a technical feature, technical solution or beneficial effect in this specification does not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions and beneficial effects described in the present embodiment can also be combined in any appropriate manner. Those skilled in the art will understand that the embodiment can be implemented without one or more specific technical features, technical solutions or beneficial effects of a specific embodiment. In other embodiments, additional technical features and beneficial effects can also be identified in specific embodiments that do not embody all embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 A system architecture diagram of a fisheye camera shooting and transmission system provided in an embodiment of the present application; Figure 2 A schematic diagram of a fisheye camera correction process provided in an embodiment of the present application; Figure 3 A schematic diagram of another fisheye camera correction process provided in an embodiment of the present application; Figure 4Examples of distorted and corrected images provided in the embodiments of the present application; Figure 5 A schematic structural diagram of a fisheye camera image correction device provided in an embodiment of the present application; Figure 6 A schematic diagram of the hardware structure of a fisheye camera image correction device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] In the description of this application, unless otherwise specified, " / " means "or", for example, A / B can mean A or B. "And / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, "at least one" means one or more, and "a plurality" means two or more. Words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not limit them to be necessarily different.

[0028] It should be noted that, in this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described in this application as "exemplary" or "for example" should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0029] The fisheye camera image correction provided by the embodiment of the present application can be applied to Figure 1 The fisheye camera captures the transmission system shown in the figure. Figure 1 As shown, the fisheye camera shooting and transmission system includes: an image acquisition device 101 and an electronic device 102.

[0030] Image acquisition device 101 is used to capture a distorted original image of a scene using a wide-angle fisheye lens and transmit the captured distorted image to an electronic device. Electronic device 102 is used to receive and decode the image transmitted from the image acquisition device, perform image correction operations based on a virtual camera model and optimization algorithm, and output the corrected image.

[0031] Fisheye cameras, with their uniquely wide field of view, are widely used in security surveillance, autonomous driving, panoramic imaging, and other fields. The images captured by fisheye cameras have a very large field of view, allowing them to cover a very large monitoring area. However, this wide field of view can cause distortion in the captured images, with the shapes of objects in the fisheye images significantly deviating from their actual shapes.

[0032] Mainstream methods for fisheye image correction include traditional algorithms based on physical calibration plates and intelligent algorithms based on deep learning. Methods based on grid-like or checkerboard calibration plates require manual operation of the camera to capture multi-view calibration images to calculate correction parameters. These methods are difficult to implement when the camera is in a closed environment, on an overhead platform, or inaccessible to direct view, due to their reliance on specific calibration templates and manual intervention.

[0033] In order to solve the technical problem that fisheye camera correction in the prior art requires reliance on specific calibration templates and manual intervention, which makes implementation difficult, the present application provides a fisheye camera image correction method, which includes: like Figure 2 As shown, the fisheye camera image correction method provided by the embodiment of the present application includes: obtaining a distorted image to be corrected, and determining the pixel coordinates of each pixel in the distorted image in the distorted image; mapping the pixel coordinates into a three-dimensional coordinate system to obtain the three-dimensional coordinates of each pixel in the three-dimensional coordinate system; projecting each of the three-dimensional coordinates into a preset corrected image to determine the corrected coordinates corresponding to each pixel in the corrected image; and mapping the pixel value of each pixel to the corresponding corrected coordinates to obtain a corrected image.

[0034] Step 201: Obtain a distorted image to be corrected, and determine the pixel coordinates of each pixel in the distorted image.

[0035] The distorted image is the original distorted image directly obtained by the fisheye camera shooting transmission system; the pixel coordinates are the position identifier of each pixel point in the image in the image coordinate system.

[0036] In one implementation, the coordinate system of the distorted image is established with the upper left corner of the distorted image as the origin, and horizontally to the right is The positive direction of the axis is vertically downward. Positive axis direction; any pixel in the distorted image , whose coordinates are ,in , and are the width and height of the distorted image, respectively.

[0037] Step 202: Map the pixel coordinates to a three-dimensional coordinate system to obtain the three-dimensional coordinates of each pixel in the three-dimensional coordinate system.

[0038] Among them, the three-dimensional coordinate system is a three-dimensional space established with the optical center of the virtual camera as the origin. It should be pointed out that the virtual camera is an ideal fisheye camera model constructed. Different from a real physical camera, the virtual camera constructed by this application does not need to obtain the intrinsic parameters of the real fisheye camera through physical calibration, and can adapt to the distortion characteristics of any fisheye camera. The three-dimensional coordinate system is established based on the intrinsic parameters of the virtual camera, but the construction method is the same as the spatial coordinate system of the real world, but with an independent projection matrix.

[0039] Step 203: Project each three-dimensional coordinate into a preset rectified image to determine the rectified coordinate corresponding to each pixel in the rectified image.

[0040] In a possible implementation, the process of presetting the image correction includes: obtaining the parameter matrix K of the virtual camera t , the parameter matrix satisfies:

[0041] in, 、 is the focal length of the virtual camera in the image coordinate system, is the principal point coordinate of the virtual camera, is the coordinate axis tilt factor, describing the image coordinate system Axis and The degree to which the angle between two axes deviates from a right angle.

[0042] Construct the pixel coordinate system of the rectified image and generate a blank image grid; initialize the blank rectified image, each pixel position , satisfying the following formula:

[0043] Among them, W and H are the resolutions of the corrected image, which are set according to the requirements. ) is a point on the normalized imaging plane of the virtual camera.

[0044] It should be pointed out that the normalized plane here is the instruction , fix the preset corrected image on a two-dimensional plane and satisfy the matrix calculation rules.

[0045] Step 204: Map the pixel value of each pixel to the corresponding correction coordinate to obtain a corrected image.

[0046] In some embodiments, combined Figure 2 ,like Figure 3 As shown, the process of mapping pixel coordinates to a three-dimensional coordinate system in step 202 can be specifically implemented by the following steps 301 to 304: Step 301: Obtain pixel coordinates and center pixel coordinates of the distorted image.

[0047] This process can be achieved by step 201, where the coordinates of any pixel of the distorted image are denoted as , the center pixel coordinates are marked as The specific implementation method will not be described here.

[0048] Step 302: Calculate the Euclidean distance from the pixel coordinates to the center coordinates.

[0049] Optionally, the Euclidean distance r satisfies the following formula:

[0050] Step 303: Determine the virtual incident angle , and the first angle .

[0051] Optionally, the virtual angle of incidence Satisfies the following formula:

[0052] in, is the focal length of the fisheye camera, yes The polynomial coefficients of . are the parameters obtained by iterative optimization using the gradient optimization method.

[0053] First Angle Satisfies the following formula: .

[0054] It should be pointed out that the above virtual incident angle In the constructed virtual camera model, the line connecting any point in the coordinates to the optical center of the camera is the line between the camera and the optical center of the camera. The angle between the axes reflects the distortion characteristics of the distorted image. Describes the spatial orientation of a pixel relative to the center of the image.

[0055] Step 304: According to the virtual incident angle and the first angle Calculate the virtual 3D coordinates corresponding to the pixel coordinates of the distorted image .

[0056] In one implementation, the virtual three-dimensional coordinates corresponding to the pixel coordinates of the distorted image are determined. This can be achieved by: According to the virtual camera internal parameters , the virtual three-dimensional coordinates Project it onto the preset corrected image to obtain the corrected coordinates ; Wherein, the pixel coordinates of any pixel position in the preset corrected image , satisfying the following formula:

[0057] in, represents the matrix multiplication operation, is the transposed matrix of the virtual three-dimensional coordinate P(x, y, z). is the virtual three-dimensional coordinate The intermediate homogeneous coordinates after matrix transformation and unnormalization.

[0058] In some embodiments, as Figure 3 As shown, the process of projecting the three-dimensional coordinates onto the preset correction image to obtain the correction coordinates in step 204 can be specifically implemented by the following steps 305 to 307, which are described in detail below: Step 305: Select a solid straight line target in the distorted image.

[0059] It should be noted that the aforementioned physical straight line target is a target that is actually a straight line in the real world, such as a railing.

[0060] Step 306 : Determine a projection loss value based on the pixel coordinates of the physical straight line target in the distorted image and the pixel coordinates of the physical straight line target in the corrected image.

[0061] In a possible implementation, the projection loss value includes a first loss value and a second loss value.

[0062] As an example, the first loss value Satisfies the following formula:

[0063] Wherein, N represents the number of entity straight line targets in the distorted image, and n represents the number of entity straight line targets among N entity straight line targets. A solid straight line target; The i-th pixel coordinate of a solid straight line target in the distorted image is marked as U n,i ( u n,i， v n,i ), the rectified pixel obtained by mapping the i-th pixel coordinate to the preset rectified image is recorded as U t n,i ( u t n,i， v t n,i ); are the indices of three different pixel points in the same physical linear target, used to traverse different physical linear targets; C is a set of elements consisting of three different coordinates randomly selected from the pixel coordinates of each linear target in the rectified image, ||C|| represents the number of elements in set C, || ||2 means The L2 norm of .

[0064] As an example, the second loss value Satisfies the following formula:

[0065] Wherein, Q represents the total number of categories of straight line targets in the distorted image classified according to the actual parallel relationship, Indicates the first n The parallel solid straight line target of the group; n The coordinates of any two pixels in the parallel solid straight line target are marked as U n,i1 ( u n,i1 ,v n,i1 ), U n,i2 ( u n,i2 ,v n,i2 ), the second corrected pixel coordinates obtained by mapping the two pixel coordinates to the preset corrected image are marked as U t n,i1 ( u t n,i1 ,v t n,i1 ), U t n,i2 ( u t n,i2 ,v t n,i2 ); The indexes of two pixels in the nth group of parallel solid line targets in the same parallel category; For The indexes of two pixels on another line in the same set of parallel solid line targets; It means that from each linear target in each category, any two point coordinates are taken from the rectified image pixels to form a set of several point pairs, and the point pairs are combined into a set.

[0066] Step 307: Based on the projection loss value, adjust the initial correction coordinates corresponding to each pixel in the corrected image to determine the correction coordinates corresponding to each pixel in the corrected image.

[0067] As an example, the gradient optimization method iteratively optimizes Satisfies the following formula:

[0068] in, ; for .

[0069] It should be noted that the above-mentioned solid straight line targets are actually straight line targets in the real world, such as railings. As the virtual angle of incidence Parameters.

[0070] In one possible implementation, steps 201 to 203 are executed to determine the corrected image, and the pixel coordinates in the distorted image taken by the fisheye camera are filled into the corresponding positions of the corrected image using the nearest neighbor interpolation or bilinear interpolation algorithm. After completing the traversal, the corrected image can be obtained. Figure 4 As shown in FIG, the distorted image captured by the fisheye camera is output after being corrected by the fisheye camera image correction method proposed in this application.

[0071] In the embodiment of the present application, the distorted image taken by the fisheye camera is mapped to a virtual three-dimensional coordinate system and then projected to a two-dimensional coordinate system. In combination with the method of optimizing parameters using an iterative algorithm, the technical problem that the traditional fisheye image correction method requires reliance on specific calibration templates and manual intervention, which makes implementation difficult, is solved.

[0072] The above mainly introduces the scheme of the embodiment of the present application from the perspective of device implementation. It is understandable that each device, for example, a fisheye camera image correction device, includes at least one of the hardware structure and software modules corresponding to each function in order to realize the above functions. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0073] The embodiment of the present application can divide the functional units of the fisheye camera image correction device according to the above method example. For example, each functional unit can be divided according to each function, or two or more functions can be integrated into one processing unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. It should be noted that the division of units in the embodiment of the present application is schematic and is only a logical functional division. There may be other division methods in actual implementation.

[0074] In the case of an integrated unit, Figure 5 A possible structural diagram of the fisheye camera image correction device (denoted as communication device 50 ) involved in the above embodiment is shown. The communication device 50 includes a processing unit 501 and a communication unit 502 , and may further include a storage unit 503 . Figure 5 The structural schematic diagram shown can be used to illustrate the structure of the fisheye camera image correction device involved in the above embodiments.

[0075] when Figure 5 The structural schematic diagram shown is used to illustrate the structure of the fisheye camera image correction device involved in the above-mentioned embodiment. The processing unit 501 is used to control and manage the operation of the fisheye camera image correction device, the communication unit 502 is used for the fisheye camera image correction device to communicate with other devices, and the storage unit 503 is used to store the program code and data of the fisheye camera image correction device.

[0076] For example, the communication unit 502 is configured to obtain a distorted image to be corrected.

[0077] The processing unit 502 is configured to determine the pixel coordinates of each pixel in the distorted image; map the pixel coordinates into a three-dimensional coordinate system to obtain the three-dimensional coordinates of each pixel in the three-dimensional coordinate system; project each of the three-dimensional coordinates into a preset corrected image to determine the corrected coordinates corresponding to each pixel in the corrected image; and map the pixel value of each pixel into the corresponding corrected coordinates to obtain a corrected image.

[0078] In a possible implementation, the processing unit 502 is further configured to obtain pixel coordinates of the distorted image. and the center pixel coordinates ;calculate arrive Euclidean distance ; Determine the virtual angle of incidence According to the virtual incident angle Calculate the virtual three-dimensional coordinates P(x, y, z) corresponding to the pixel coordinates; project the virtual three-dimensional coordinates P(x, y, z) into the preset corrected image to obtain the pixel coordinates of any pixel position in the preset corrected image ; Determine the projection loss value based on the pixel coordinate points of the physical straight line target in the distorted image and the pixel coordinate points of the physical straight line target in the corrected image; Based on the projection loss value, adjust the initial correction coordinates corresponding to each pixel in the corrected image to determine the correction coordinates corresponding to each pixel in the corrected image.

[0079] In one possible implementation, the Euclidean distance Satisfies the following formula:

[0080] in, is the pixel coordinate of the distorted image; is the center pixel coordinate.

[0081] In one possible implementation, the virtual angle of incidence is , satisfying the following formula:

[0082] in, is the focal length of the fisheye camera, are the optimized polynomial coefficients.

[0083] In one possible implementation, the pixel coordinates of any pixel position in the rectified image are preset Satisfies the following formula:

[0084] in, represents the matrix multiplication operation, is the transposed matrix of the virtual three-dimensional coordinate P(x, y, z); is the virtual three-dimensional coordinate The intermediate homogeneous coordinates after matrix transformation and unnormalization.

[0085] The processing unit 501 may be a processor or a controller, and the communication unit 502 may be a communication interface, a transceiver, a transceiver, a transceiver circuit, a transceiver device, etc. The communication interface is a general term and may include one or more interfaces. The storage unit 503 may be a memory. When the communication device 50 is a chip, the processing unit 501 may be a processor or a controller, and the communication unit 502 may be an input interface and / or output interface, a pin or a circuit, etc. The storage unit 503 may be a storage unit within the chip (e.g., a register, a cache, etc.), or a storage unit located outside the chip (e.g., a read-only memory (ROM), a random access memory (RAM), etc.).

[0086] Among them, the communication unit can also be called a transceiver unit. The antenna and control circuit with transceiver functions in the communication device 50 can be regarded as the communication unit 502 of the communication device 50, and the processor with processing function can be regarded as the processing unit 501 of the communication device 50. Optionally, the device used to implement the receiving function in the communication unit 502 can be regarded as the communication unit, and the communication unit is used to perform the receiving steps in the embodiment of the present application. The communication unit can be a receiver, a receiver, a receiving circuit, etc. The device used to implement the sending function in the communication unit 502 can be regarded as a sending unit, and the sending unit is used to perform the sending steps in the embodiment of the present application. The sending unit can be a transmitter, a transmitter, a sending circuit, etc.

[0087] Figure 5 If the integrated units are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The storage medium for storing computer software products includes various media that can store program codes, such as USB flash drives, mobile hard drives, read-only memories, random access memories, magnetic disks or optical disks.

[0088] Figure 5 A unit in a can also be called a module, for example, a processing unit can be called a processing module.

[0089] The embodiment of the present application also provides a hardware structure diagram of a fisheye camera image correction device (denoted as a communication device 60), see Figure 6The communication device 60 includes a processor 601 and, optionally, a memory 602 connected to the processor 601.

[0090] In the first possible implementation, see Figure 6 The communication device 60 further includes a transceiver 603. The processor 601, the memory 602, and the transceiver 603 are connected via a bus. The transceiver 603 is used to communicate with other devices or a communication network. Optionally, the transceiver 603 may include a transmitter and a receiver. The device used to implement the receiving function in the transceiver 603 can be considered a receiver, and the receiver is used to perform the receiving steps in the embodiments of the present application. The device used to implement the transmitting function in the transceiver 603 can be considered a transmitter, and the transmitter is used to perform the transmitting steps in the embodiments of the present application.

[0091] Based on the first possible implementation, Figure 6 The structural schematic diagram shown can be used to illustrate the structure of the fisheye camera image correction device involved in the above embodiments.

[0092] in, Figure 6 The system chip in the fisheye camera image correction device can also be illustrated. In this case, the actions performed by the fisheye camera image correction device can be implemented by the system chip. The specific actions performed can be found above and will not be repeated here.

[0093] During implementation, each step of the method provided in this embodiment can be completed by hardware integrated logic circuits in a processor or by software instructions. The steps of the method disclosed in the embodiments of this application can be directly implemented as execution by a hardware processor, or as a combination of hardware and software modules in a processor.

[0094] The processor in this application may include, but is not limited to, at least one of the following: a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a microcontroller unit (MCU), or an artificial intelligence processor, among other types of computing devices that run software. Each computing device may include one or more cores for executing software instructions to perform operations or processing. The processor may be a standalone semiconductor chip or integrated into a semiconductor chip with other circuits. For example, it may form a system-on-chip (SoC) with other circuits (such as a codec circuit, a hardware acceleration circuit, or various bus and interface circuits). Alternatively, it may be integrated into an ASIC as a built-in processor. The ASIC with the integrated processor may be packaged separately or together with other circuits. In addition to the core for executing software instructions to perform operations or processing, the processor may further include necessary hardware accelerators, such as a field programmable gate array (FPGA), a PLD (programmable logic device), or logic circuits that implement specialized logic operations.

[0095] The memory in the embodiments of the present application may include at least one of the following types: read-only memory (ROM) or other types of static storage devices that can store static information and instructions, random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, and electrically erasable programmable read-only memory (EEPROM). In some scenarios, the memory may also be a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto.

[0096] An embodiment of the present application also provides a computer-readable storage medium, comprising instructions, which, when executed on a computer, enables the computer to execute any of the above methods.

[0097] An embodiment of the present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the above methods.

[0098] An embodiment of the present application also provides a chip, which includes a processor and an interface circuit, the interface circuit is coupled to the processor, the processor is used to run a computer program or instruction to implement the above method, and the interface circuit is used to communicate with other modules outside the chip.

[0099] In the above embodiments, all or part of the embodiments can be implemented using software, hardware, firmware, or any combination thereof. When implemented using a software program, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media. Available media may be magnetic media (eg, floppy disks, hard disks, magnetic tapes), optical media (eg, DVDs), or semiconductor media (eg, solid state disks (SSDs)).

[0100] Although the present application is described herein in conjunction with various embodiments, in the process of implementing the claimed application, those skilled in the art may understand and implement other variations of the disclosed embodiments by reviewing the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple situations. A single processor or other unit may implement several functions listed in the claims. The fact that certain measures are recorded in different dependent claims does not mean that these measures cannot be combined to produce good results.

[0101] Although the present application has been described with reference to specific features and embodiments thereof, it is apparent that various modifications and combinations may be made thereto without departing from the spirit and scope of the present application. Accordingly, this specification and the drawings are merely illustrative of the present application as defined by the appended claims and are deemed to cover any and all modifications, variations, combinations or equivalents within the scope of the present application. Obviously, those skilled in the art may make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, the present application is intended to include such modifications and variations as fall within the scope of the claims of the present application and their equivalents.

Claims

1. A fisheye camera image correction method, characterized in that: include: Acquire a distorted image to be corrected, and determine the pixel coordinates of each pixel in the distorted image in the distorted image; Mapping the pixel coordinates into a three-dimensional coordinate system to obtain the three-dimensional coordinates of each pixel in the three-dimensional coordinate system; Projecting each of the three-dimensional coordinates into a preset rectified image to determine the rectified coordinate corresponding to each pixel in the rectified image; The pixel value of each pixel is mapped to the corresponding correction coordinate to obtain a corrected image.

2. The method according to claim 1, characterized in that The process of mapping the pixel coordinates of the distorted image to a three-dimensional coordinate system includes: Get the pixel coordinates of the distorted image and the center pixel coordinates ; calculate arrive Euclidean distance : ; Determine the virtual angle of incidence ; According to the virtual incident angle and stated , calculate the virtual three-dimensional coordinates P(x, y, z) corresponding to the pixel coordinates, and the three-dimensional coordinates satisfy the following calculation formula: 。 3. The method according to claim 2, characterized in that The virtual incident angle , satisfying the following formula: in, is the focal length of the fisheye camera, are the optimized polynomial coefficients.

4. The method according to claim 1, wherein The process of mapping 3D coordinates into rectified images includes: According to the virtual camera internal parameters , project the virtual three-dimensional coordinates P(x, y, z) onto the preset rectified image to obtain the rectified coordinates ; Wherein, the pixel coordinates of any pixel position in the preset corrected image , satisfying the following formula: in, represents the matrix multiplication operation, is the transposed matrix of the virtual three-dimensional coordinate P(x, y, z); is the virtual three-dimensional coordinate The intermediate homogeneous coordinates after matrix transformation and unnormalization.

5. The method according to any one of claims 1 to 4, characterized in that Projecting each of the three-dimensional coordinates into a preset rectified image to determine the rectified coordinate corresponding to each pixel in the rectified image includes: Projecting each of the three-dimensional coordinates into a preset rectified image to obtain an initial rectified coordinate corresponding to each pixel in the rectified image; Acquire a physical straight line target in the distorted image, and determine a projection loss value based on pixel coordinates of the physical straight line target in the distorted image and pixel coordinates of the physical straight line target in the corrected image; Based on the projection loss value, the initial rectified coordinate corresponding to each pixel in the rectified image is adjusted to determine the rectified coordinate corresponding to each pixel in the rectified image.

6. The method according to claim 5, characterized in that The projection loss value includes a first loss value, and determining the projection loss value based on the pixel coordinate point of the physical straight line target in the distorted image and the pixel coordinate point of the physical straight line target in the corrected image includes: Get the coordinates of any three pixel points on the same straight line target; Determining three first corrected pixel coordinate points corresponding to the three pixel coordinate points in the corrected image; A first loss value is determined based on the three first corrected pixel coordinate points, where the first loss value is used to represent a deviation value between a line connecting the three first corrected pixel coordinate points and a straight line.

7. The method according to claim 5, characterized in that The projection loss value further includes a second loss value, and determining the projection loss value based on the pixel coordinate point of the physical straight line target in the distorted image and the pixel coordinate point of the physical straight line target in the corrected image includes: Acquire parallel physical straight line targets in the distorted image, and acquire at least two coordinate points of each parallel physical straight line target; Determining second rectified pixel coordinates corresponding to two coordinate points of each parallel solid straight line object in the rectified image; A second loss value of the parallel physical straight line object is calculated based on the second corrected pixel coordinates, where the second loss value is used to represent a parallel deviation value after the parallel physical straight line object is projected onto the preset corrected image.

8. The method according to claim 6, characterized in that The first loss Satisfies the following formula: Wherein, N represents the number of entity straight line targets in the distorted image, and n represents the number of entity straight line targets among N entity straight line targets. A solid straight line target; The i-th pixel coordinate of a solid straight line target in the distorted image is marked as U n,i ( u n,i , v n,i ), the rectified pixel obtained by mapping the i-th pixel coordinate to the preset rectified image is recorded as U t n,i ( u t n,i ,v t n,i ); are the indices of three different pixel points in the same physical linear target, used to traverse different physical linear targets; C is a set of elements consisting of three different coordinates randomly selected from the pixel coordinates of each linear target in the rectified image, ||C|| represents the number of elements in set C, || ||2 means The L2 norm of .

9. The method according to claim 7, characterized in that The second loss Satisfy the following calculation formula: Wherein, Q represents the total number of categories of straight line targets in the distorted image classified according to the actual parallel relationship, Indicates the first n The parallel solid straight line target of the group; n The coordinates of any two pixels in the parallel solid straight line target are marked as U n,i1 ( u n,i1 ,v n,i1 ), U n,i2 ( u n,i2 ,v n,i2 ), the second corrected pixel coordinates obtained by mapping the two pixel coordinates to the preset corrected image are marked as U t n,i1 ( u t n,i1 ,v t n,i1 ), U t n,i2 ( u t n,i2 ,v t n,i2 ); The indexes of two pixels in the nth group of parallel solid line targets in the same parallel category; For The indexes of two pixels on another line in the same set of parallel solid line targets; It means that from each linear target in each category, any two point coordinates are taken from the rectified image pixels to form a set of several point pairs, and the point pairs are combined into a set.

10. The method according to claim 5, characterized in that The adjustment method includes: Iterative optimization using gradient optimization method , Satisfies the following formula: in, ; for .

Images