Image correction method and device, electronic equipment, storage medium and vehicle

Abstract

The application provides an image correction method and device, electronic equipment, storage medium and vehicle. The method comprises the following steps: acquiring an original image; taking the original image as a texture map and pasting it to a set arc surface; wherein the plane in which the straight lines at the two side edges of the set arc surface are located is parallel to the image plane in which the original image is located, and the intersection line between the set arc surface and the tangent plane perpendicular to the image plane is an arc; and projecting the pasted set arc surface according to virtual camera parameters and target projection parameters to obtain a corrected target image. Thus, the method does not need to calibrate each camera, takes each frame of video content as a texture map, pastes it on a set arc surface, and then projects the pasted set arc surface to obtain a corrected target image, so that the workload is reduced, the efficiency is improved, the relative normality of the display of objects close to the user is ensured under the condition of preserving a large field of view, and the discomfort of the user is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to an image correction method and device, electronic equipment, storage medium and vehicle. BACKGROUND

[0002] The field of view angle of a traditional camera is relatively small, so there is inevitably a large blind area, which makes it difficult to meet actual requirements. A fisheye camera is a lens with a field of view angle greater than 120 degrees. Its large field of view makes it widely used in many fields of life, but the image captured by the fisheye camera has serious distortion, which makes people feel very uncomfortable. Therefore, it is necessary to restore the image captured by the fisheye camera to an image that conforms to human viewing, that is, to correct the distortion of the fisheye image.

[0003] In related technologies, when correcting the distortion of a fisheye image, the fisheye camera often needs to be calibrated in advance. If each fisheye camera is calibrated, it will result in a large amount of work and low efficiency. Moreover, according to the distortion correction effect of related technologies, under the premise of retaining a large field of view, objects close to the vehicle will appear abnormally large, and users will feel uncomfortable when viewing. SUMMARY

[0004] The present application aims to at least partially solve one of the technical problems in the related art.

[0005] To this end, a first object of the present application is to propose an image correction method, which does not need to calibrate the fisheye camera in advance, but only needs to paste each frame of video content as a texture map on a set arc surface, and project the set arc surface after pasting according to target virtual camera parameters and target projection parameters, to obtain a corrected target image. This can reduce the amount of work, improve efficiency, and make the display of objects close to the vehicle relatively normal under the condition of retaining a large angle of view, thereby reducing the discomfort of users when viewing.

[0006] A second object of the present application is to propose an image correction device.

[0007] A third object of the present application is to propose an electronic device.

[0008] A fourth object of the present application is to propose a non-transitory computer-readable storage medium.

[0009] A fifth object of the present application is to propose a vehicle.

[0010] A sixth object of the present application is to propose a computer program product.

[0011] To achieve the above object, the first aspect of the present application provides a method, comprising: obtaining an original image; and pasting the original image as a texture map to a set arc surface, wherein a plane in which straight lines at two side edges of the set arc surface are located is parallel to an image plane in which the original image is located, and an intersection between the set arc surface and a tangent plane perpendicular to the image plane is an arc; and projecting the set arc surface after pasting according to target virtual camera parameters and target projection parameters to obtain a corrected target image.

[0012] According to the image correction method provided by the embodiment of the present application, the original image is obtained first, and the original image is pasted as a texture map to a set arc surface, wherein a plane in which straight lines at two side edges of the set arc surface are located is parallel to an image plane in which the original image is located, and an intersection between the set arc surface and a tangent plane perpendicular to the image plane is an arc, and then the set arc surface after pasting is projected according to target virtual camera parameters and target projection parameters to obtain a corrected target image. Thus, the method does not need to calibrate a camera in advance, only needs to paste each frame of video content as a texture map to a set arc surface, and projects the set arc surface after pasting according to target virtual camera parameters and target projection parameters to obtain a corrected target image, so that the workload can be reduced, the efficiency can be improved, and the relative normality of display of objects close to the user can be ensured in the case of a large viewing angle, thereby reducing the discomfort of the user.

[0013] In addition, the image correction method provided by the first aspect of the present application can have the following additional technical features.

[0014] According to an embodiment of the present application, after the original image to be corrected is obtained, the method further comprises: determining a radius of a visible region in the original image; and determining a radius of the set arc surface according to the radius of the visible region.

[0015] According to an embodiment of the present application, the pasting of the original image as a texture map to a set arc surface comprises: determining an origin of a three-dimensional coordinate system in which the set arc surface is located according to a center of a visible region in the original image, wherein an X axis in the three-dimensional coordinate system is parallel to the image plane, and a Y axis is parallel to a straight line coinciding with the set arc surface and perpendicular to the X axis; determining a texture coordinate corresponding to a target point on the set arc surface according to X axis and Y axis coordinates of the target point in the three-dimensional coordinate system; and pasting a pixel unit having the texture coordinate in a texture coordinate system in which the original image is located to the target point on the set arc surface according to the texture coordinate corresponding to the target point.

[0016] According to one of the embodiments of the present application, the method further comprises: determining the target optical axis direction of the virtual camera according to the direction parameter in the target virtual camera parameter; determining the target field of view angle of the virtual camera according to the angle parameter in the target projection parameter; and projecting the mapped set arc surface to the virtual camera along the target optical axis direction based on the target field of view angle to obtain the corrected target image.

[0017] According to one of the embodiments of the present application, the method further comprises: determining the target optical axis direction of the virtual camera according to the direction parameter in the target virtual camera parameter; determining the target field of view angle of the virtual camera according to the angle parameter in the target projection parameter; and projecting the mapped set arc surface to the virtual camera along the target optical axis direction based on the target field of view angle to obtain the corrected target image.

[0018] According to one of the embodiments of the present application, the method further comprises: determining the target optical axis direction of the virtual camera according to the direction parameter in the target virtual camera parameter; determining the target field of view angle of the virtual camera according to the angle parameter in the target projection parameter; and projecting the mapped set arc surface to the virtual camera along the target optical axis direction based on the target field of view angle to obtain the corrected target image.

[0019] According to one of the embodiments of the present application, the method further comprises: determining the target virtual camera parameter and the target projection parameter according to the model of the fisheye camera that collects the original image.

[0020] According to one of the embodiments of the present application, the method further comprises: projecting the mapped set arc surface according to a plurality of candidate virtual camera parameters and a plurality of candidate projection parameters to obtain reference images; determining a selected image from the reference images corresponding to each of the candidate virtual camera parameters and the candidate projection parameters in response to a user operation; and taking the candidate virtual camera parameter and the candidate projection parameter corresponding to the selected image as the target virtual camera parameter and the target projection parameter, respectively.

[0021] To achieve the above object, the second aspect of the present application provides an image correction device, comprising: an acquisition module, configured to acquire an original image; a mapping module, configured to map the original image as a texture map to a set arc surface; wherein a straight line in the set arc surface at both side edges is parallel to an image plane where the original image is located, and an intersection between the set arc surface and a tangent plane perpendicular to the image plane is an arc; and a first projection module, configured to project the set arc surface after mapping according to target virtual camera parameters and target projection parameters to obtain a corrected target image.

[0022] According to the image correction device provided by the embodiment of the present application, the original image is acquired by the acquisition module, the original image is mapped as a texture map to the set arc surface by the mapping module, wherein the straight line in the set arc surface at both side edges is parallel to the image plane where the original image is located, and the intersection between the set arc surface and the tangent plane perpendicular to the image plane is an arc, and the set arc surface after mapping is projected according to the target virtual camera parameters and the target projection parameters by the first projection module to obtain the corrected target image. Thus, the device does not need to calibrate the camera in advance, only needs to map each frame of video content as a texture map to the set arc surface, and projects the set arc surface after mapping according to the target virtual camera parameters and the target projection parameters to obtain the corrected target image, which can reduce the workload, improve the efficiency, and make the near objects displayed relatively normal under the condition of preserving a large viewing angle, thereby reducing the discomfort of the user.

[0023] In addition, the image correction device provided by the second aspect of the present application can have the following additional technical features:

[0024] According to an embodiment of the present application, the image correction device further comprises: a first determination module, configured to determine the radius of a visible area in the original image after acquiring the original image to be corrected; and a second determination module, configured to determine the radius of the set arc surface according to the radius of the visible area.

[0025] According to an embodiment of the present application, the mapping module comprises: a first determining unit configured to determine an origin of a three-dimensional coordinate system in which the set arc-shaped surface is located according to a center of a visible region in the original image; wherein an X-axis in the three-dimensional coordinate system is parallel to the image plane, and a Y-axis is a straight line parallel to the set arc-shaped surface and perpendicular to the X-axis; a second determining unit configured to determine corresponding texture coordinates according to X-axis and Y-axis coordinates of a target point on the set arc-shaped surface in the three-dimensional coordinate system; and a mapping unit configured to map a pixel unit having the texture coordinates in a texture coordinate system in which the original image is located to the target point according to the texture coordinates.

[0026] According to an embodiment of the present application, the second determining unit comprises: a translation sub-unit configured to translate X-axis and Y-axis coordinates (X, Y) of the target point in the three-dimensional coordinate system according to coordinates (x0, y0) of the center of the visible region in a texture coordinate system in which the original image is located; and a processing sub-unit configured to normalize the translated coordinates according to a resolution of the original image to obtain the texture coordinates corresponding to the target point.

[0027] According to an embodiment of the present application, the first projection module comprises: a third determining unit configured to determine a target optical axis direction of the virtual camera according to a direction parameter in the target virtual camera parameter; a fourth determining unit configured to determine a target field of view angle of the virtual camera according to an angle parameter in the target projection parameter; and a first projection unit configured to project the mapped set arc-shaped surface to the virtual camera along the target optical axis direction based on the target field of view angle to obtain a corrected target image.

[0028] According to an embodiment of the present application, the first projection unit comprises: a first determining sub-unit configured to determine a virtual camera matrix according to the target optical axis direction of the virtual camera; a second determining sub-unit configured to determine a projection matrix between an imaging coordinate system of the virtual camera and a three-dimensional coordinate system in which the arc-shaped surface is located according to the target field of view angle of the virtual camera; and a projection sub-unit configured to project the mapped set arc-shaped surface to the virtual camera along the target optical axis direction based on the virtual camera matrix and the projection matrix to obtain the corrected target image.

[0029] According to an embodiment of the present application, the image correction device further comprises: a third determining module configured to determine the target virtual camera parameter and the target projection parameter according to a model of a fisheye camera from which the original image is collected.

[0030] According to one embodiment of the present application, the image correction device described above further comprises: a second projection module, configured to project the set arc surface after the mapping according to the plurality of candidate virtual camera parameters and the plurality of candidate projection parameters to obtain reference images; a fourth determination module, configured to determine a selected image from the reference images corresponding to each of the candidate virtual camera parameters and each of the candidate projection parameters in response to a user operation; and a setting module, configured to set the candidate virtual camera parameter corresponding to the selected image as the target virtual camera parameter and set the candidate projection parameter corresponding to the selected image as the target projection parameter.

[0031] To achieve the above object, the third aspect of the present application provides an electronic device, comprising: a processor and a memory; wherein the processor runs a program corresponding to executable program code stored in the memory to implement the image correction method described above.

[0032] The electronic device of the embodiment of the present application can reduce workload, improve efficiency, and make the display of nearby objects relatively normal while preserving a large viewing angle by implementing the image correction method described above without calibrating the camera in advance, by only mapping each frame of video content as a texture map on a set arc surface, and projecting the set arc surface after the mapping according to target virtual camera parameters and target projection parameters to obtain a corrected target image.

[0033] To achieve the above object, the fourth aspect of the present application provides a non-transitory computer readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the image correction method described above.

[0034] The non-transitory computer readable storage medium of the embodiment of the present application can reduce workload, improve efficiency, and make the display of nearby objects relatively normal while preserving a large viewing angle by implementing the image correction method described above without calibrating the camera in advance, by only mapping each frame of video content as a texture map on a set arc surface, and projecting the set arc surface after the mapping according to target virtual camera parameters and target projection parameters to obtain a corrected target image.

[0035] To achieve the above object, the fifth aspect of the present application provides a vehicle comprising the electronic device described above.

[0036] The vehicle of the embodiment of the present application, when using the fisheye camera, does not need to calibrate the fisheye camera in advance, only needs to paste each frame of video content as a texture map on a set arc surface, and projects the set arc surface after pasting according to target virtual camera parameters and target projection parameters, so that the corrected target image can be obtained, which can reduce workload, improve efficiency, and make the display of the near object relatively normal in the case of retaining a large viewing angle, and reduce the discomfort of the user when watching.

[0037] To achieve the above object, the sixth aspect embodiment of the present application provides a computer program product, when the instruction processor in the computer program product executes, the image correction method described above is executed.

[0038] The computer program product of the embodiment of the present application, by executing the image correction method described above, does not need to calibrate the camera in advance, only needs to paste each frame of video content as a texture map on a set arc surface, and projects the set arc surface after pasting according to target virtual camera parameters and target projection parameters, so that the corrected target image can be obtained, which can reduce workload, improve efficiency, and make the display of the near object relatively normal in the case of retaining a large viewing angle, and reduce the discomfort of the user when watching.

[0039] Additional aspects and advantages of the present application will be made apparent by the following description and the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0040] The above and / or additional aspects and advantages of the present application will become apparent and be readily understood from the following description, taken in conjunction with the accompanying drawings, in which:

[0041] Figure 1 is a flowchart of the image correction method according to an embodiment of the present application;

[0042] Figure 2 is a schematic diagram of the image correction according to an embodiment of the present application;

[0043] Figure 3 is a flowchart of the image correction method according to an embodiment of the present application;

[0044] Figure 4 is a flowchart of the image correction method according to another embodiment of the present application;

[0045] Figure 5 is a flowchart of the image correction method according to still another embodiment of the present application;

[0046] Figure 6 is a block schematic diagram of the image correction device according to an embodiment of the present application. DETAILED DESCRIPTION

[0047] Embodiments of the present application are described below in detail, examples of which are shown in the accompanying drawings, in which the same or similar reference numerals represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the drawings are exemplary and are intended to explain the present application, and cannot be understood as a limitation of the present application.

[0048] The image correction method, the image correction device, the electronic device and the non-transitory computer readable storage medium of the embodiments of the present application are described below with reference to the accompanying drawings.

[0049] The field of view of a traditional camera is relatively small, and there is inevitably a large blind area when used in a vehicle-mounted vision system, which makes it difficult to meet actual requirements. A fisheye camera is a lens with a field of view greater than 120 degrees, and a camera with a field of view of about 180 degrees can meet the field of view requirement of a vehicle-mounted vision system. The large field of view of a fisheye camera makes it widely used in many fields of life, but the images taken by the fisheye camera have serious distortion, which makes people feel very uncomfortable. If we need to use the content of these distorted images, we need to correct these images to conform to people's habits. Restoring the images taken by the fisheye lens to conform to human viewing is called distortion correction.

[0050] In the related art, the fisheye image correction has a large amount of calculation, and the real-time video stream distortion correction needs to occupy a lot of resources. For embedded devices, due to the limitation of the processor frequency, it needs to spend a lot of cost to achieve real-time performance. Through the parallel computing technology opengl (Open Graphics Library, open graphics library) using GPU (Graphics Processing Unit, graphics processing unit) processing, the limitation of the processor frequency can be broken through, and high-performance computing can be realized. At present, most of the video correction algorithms based on CPU (Central Processing Unit, central processing unit) are processed, which often cannot meet the requirement of real-time correction of fisheye video or the CPU occupancy rate is very high, which has a greater impact on the operation of the whole system. The GPU has a large-scale parallel throughput architecture with multiple concurrent threads, and has strong flexibility, which can meet the application requirement of fisheye video correction. In the related art, the distortion correction of the fisheye image often needs to calibrate the fisheye camera in advance to determine the internal parameters and distortion parameters of the fisheye camera. If each fisheye camera needs to be calibrated, it will result in a large amount of work and low efficiency. Moreover, according to the distortion correction effect of the related art, under the premise of retaining a large field of view, objects close to the vehicle will appear abnormally large, and users will have an uncomfortable feeling when watching.

[0051] To this end, the embodiment of the present application provides an image correction method, which does not need to calibrate the fisheye camera in advance, only needs to take each frame of video content as a texture map and paste it on a set arc surface, and projects the pasted set arc surface according to target virtual camera parameters and target projection parameters, so as to obtain a corrected image, which can reduce workload, improve efficiency, and make the display of a near object relatively normal in the case of preserving a large viewing angle, thereby reducing the discomfort of a user.

[0052] Figure 1 is a schematic diagram of the image correction method according to the embodiment of the present application.

[0053] As Figure 1 shown, the image correction method of the embodiment of the present application comprises the following steps:

[0054] S101, obtaining an original image.

[0055] In this step, the original image is a to-be-corrected original image directly obtained by using a fisheye camera, and the shape of the to-be-corrected original image can be circular, as Figure 2 shown.

[0056] S102, taking the original image as a texture map and pasting it to a set arc surface; wherein the planes in which the straight lines at both side edges of the set arc surface are located are parallel to the image plane in which the original image is located, and the intersection line between the set arc surface and the tangent plane perpendicular to the image plane is an arc.

[0057] It should be noted that the set arc surface can be a semicylindrical arc surface, a semispherical arc surface, a semielliptical arc surface, etc.

[0058] For example, when the set arc surface is a semicylindrical arc surface, a cylindrical model or a tubular model is first established, the cylindrical model or the tubular model is cut open in the length direction, the cross section after the cutting is parallel to the image plane in which the original image is located, and then the set arc surface in the present application is formed. In the present application, each frame of video content captured by the fisheye camera is taken as a texture map and pasted to the set arc surface, as shown by the dotted arrow. Figure 2

[0059] S103, projecting the pasted set arc surface according to target virtual camera parameters and target projection parameters to obtain a corrected target image.

[0060] ​In this step, the target virtual camera parameter can include the position of the virtual camera, the orientation of the virtual camera head and the direction of the virtual camera sight line; the target projection parameter can include the angle of the field of view of the view volume, the width-to-height ratio of the view volume, the distance from the observer to the nearest arc surface of the view volume and the distance from the observer to the farthest arc surface of the view volume. It should be noted that, when performing correction for the first time for the imaging of the fisheye camera of this model, the mapped set arc surface can be projected according to the plurality of candidate virtual camera parameters and the plurality of candidate projection parameters to obtain reference images, and the user selects a suitable image from the plurality of reference images, at this time, the candidate camera parameter and the candidate projection parameter corresponding to the image selected by the user are taken as the target virtual camera parameter and the target projection parameter respectively; when performing correction for the imaging of the fisheye camera of this model for the first time, the model of the fisheye camera is acquired, a correspondence table of the model of the fisheye camera, the target virtual camera parameter and the target projection parameter is stored in advance, and the target virtual camera parameter and the target projection parameter can be directly determined according to the model of the fisheye camera by looking up the table.

[0061] After the target virtual camera parameter and the target projection parameter are acquired, the mapped set arc surface is rendered onto the screen according to the target virtual camera parameter and the target projection parameter, so as to obtain a corrected target image, so that the user can watch the corrected video on the screen.

[0062] Therefore, the image correction method of the embodiment of the present application does not need to calibrate the fisheye camera in advance, directly establishes the set arc surface model, maps each frame of video content as a texture map on the set arc surface, and projects each frame of the mapped set arc surface by changing the sight line direction and the field of view angle of the virtual camera at the center of the set arc surface, so as to obtain a target video with a de-distortion effect. In this way, the display of the object in the near place can be relatively normal, and the discomfort of the user when watching can be reduced while the large viewing angle is retained.

[0063] Figure 3 is a flowchart of the image correction method according to an embodiment of the present application.

[0064] As shown in Figure 3 , the image correction method of the embodiment of the present application includes the following steps:

[0065] S301, acquiring an original image.

[0066] In this step, the original image is a to-be-corrected original image directly captured by using the fisheye camera, and the edge of the to-be-corrected original image can be black, as shown in Figure 2 .

[0067] S302, determining the radius of the visible region in the original image.

[0068] In this step, the visible area is a closed area including black edges that can be seen by human eyes, and the closed area is a circle, and the radius r of the visible area is the radius of the circular area.

[0069] S303, determining the radius of the set arc surface according to the radius of the visible area.

[0070] In this step, the radius R of the set arc surface is defined as the radius r of the visible area.

[0071] S304, determining the origin of the three-dimensional coordinate system in which the set arc surface is located according to the center of the visible area in the original image; wherein the X-axis in the three-dimensional coordinate system is parallel to the image plane, and the Y-axis is parallel to the straight line coinciding with the set arc surface and perpendicular to the X-axis.

[0072] In this step, the pixel coordinates of the center of the original image are (x0, y0, 0), and the radius is r, which is in pixels.

[0073] Establishing the three-dimensional coordinate system in which the set arc surface is located and the set arc surface: taking the center of the original image as the coordinate origin, the right side of the original image as the positive direction of the X-axis, the lower side of the original image as the positive direction of the Y-axis, and the vertical direction of the original image as the positive direction of the Z-axis, a three-dimensional coordinate system is established, and the unit is 1 pixel. Taking the center of the original image as the center of the semicylinder and the Y-axis direction as the central axis, a semicylinder with a radius of r is established, and the arc surface of the semicylinder is the set arc surface.

[0074] S305, determining the corresponding texture coordinates according to the X-axis and Y-axis coordinates of the target point on the set arc surface in the three-dimensional coordinate system.

[0075] In this step, according to the coordinates (x0, y0) of the center of the visible area in the texture coordinate system (the texture coordinate system is the coordinate system in which the x-axis and y-axis are as shown in Figure 2 In this step, the X-axis and Y-axis coordinates (X, Y) of the target point on the set arc surface in the three-dimensional coordinate system are translated according to the coordinates (x0, y0) of the center of the visible area in the texture coordinate system (the texture coordinate system is the coordinate system in which the x-axis and y-axis are as shown in

[0076] In order to enhance the texture of the target image, the normalized coordinates can be obtained by normalizing the translated coordinates. For example, w and h are the resolutions of the original image in the X-axis and Y-axis, respectively. The normalized coordinates are obtained by dividing the X-axis value of the translated coordinates by the resolution w of the original image in the X-axis, and dividing the Y-axis value of the translated coordinates by the resolution h of the original image in the Y-axis.

[0077] S306, according to the texture coordinates, the pixel unit with the texture coordinates in the texture coordinate system in which the original image is located is pasted to the target point.

[0078] That is, the pixel unit corresponding to each texture coordinate in the texture coordinate system in the visual region of the original image is pasted to the corresponding position of the set arc surface.

[0079] S307, according to the direction parameter in the target virtual camera parameter, the target optical axis direction of the virtual camera is determined.

[0080] The target virtual camera parameter can include the position of the virtual camera, the orientation of the head of the virtual camera, and the direction of the line of sight of the virtual camera. The virtual camera matrix can be constructed according to the target virtual camera parameter, that is, the filling function gluLookAt(eye, at, up) is filled, wherein the position eye of the fisheye camera is the origin of the virtual camera imaging coordinate system, that is, the position of the virtual camera; the line of sight at of the fisheye camera is the optical axis direction of the virtual camera, that is, the direction of the line of sight of the virtual camera, the direction of the line of sight at of the fisheye camera and the included angle a with the X axis and the included angle b with the Z axis; the head up of the fisheye camera faces the negative direction of the Y axis, that is, the orientation of the head of the virtual camera (that is, the direction of the shutter of the virtual camera).

[0081] It should be noted that after the pixel unit with the texture coordinates in the texture coordinate system in which the original image is located is pasted to the target point, the filling function is used to fill the image blank area of the set arc surface after pasting.

[0082] S308, according to the angle parameter in the target projection parameter, the target field of view angle of the virtual camera is determined.

[0083] The target projection parameter can include the angle of the field of view of the view volume, the aspect ratio of the view volume, the distance from the observer to the nearest arc surface of the view volume, and the distance from the observer to the farthest arc surface of the view volume. The projection matrix gluPerspective(fovy, aspect, zNear, zFar) can be established through the projection parameter, wherein fovy is the angle of the field of view of the view volume; aspect is the aspect ratio of the view volume; zNear is the distance from the observer to the nearest arc surface of the view volume; zFar is the distance from the observer to the farthest arc surface of the view volume. The target field of view angle of the virtual camera can be determined according to the angle fovy of the field of view of the view volume.

[0084] S309, based on the target field of view angle, the set arc surface after pasting is projected to the virtual camera along the target optical axis direction, to obtain the corrected target image.

[0085] In this step, after the target visual field angle and the target optical axis direction are acquired, a virtual camera matrix gluLookAt(eye, at, up) is determined according to the target optical axis direction of the virtual camera, and a projection matrix gluPerspective(fovy, aspect, zNear, zFar) between the imaging coordinate system of the virtual camera and the three-dimensional coordinate system in which the arc-shaped surface is located is determined according to the target visual field angle of the virtual camera, and then the mapped set arc-shaped surface is projected to the virtual camera along the target optical axis direction of the virtual camera based on the virtual camera matrix and the projection matrix, so as to obtain the corrected target image.

[0086] Therefore, the image correction method of the embodiment of the present application can directly establish a set arc-shaped surface model without calibrating the fisheye camera in advance, map each frame of video content as a texture map on the set arc-shaped surface, and project each mapped set arc-shaped surface by changing the sight direction and the field angle of the virtual camera at the center of the set arc-shaped surface, so as to obtain a target video with a distortion-removed effect. In this way, the video effect of a relatively normal object in the near distance can be displayed while the large visual angle is retained, and the discomfort of the user during watching is reduced.

[0087] Figure 4 is a flowchart of the image correction method according to another embodiment of the present application.

[0088] As shown in Figure 4 , the image correction method of the embodiment of the present application comprises the following steps:

[0089] S401, acquiring an original image.

[0090] S402, mapping the original image as a texture map on a set arc-shaped surface; wherein the planes in which the straight lines at the two side edges of the set arc-shaped surface are located are parallel to the image plane in which the original image is located, and the intersection line between the set arc-shaped surface and the tangent plane perpendicular to the image plane is an arc.

[0091] S403, determining target virtual camera parameters and target projection parameters according to the model of the fisheye camera from which the original image is collected.

[0092] In this embodiment, a correspondence table of different models of fisheye cameras and target virtual camera parameters and target projection parameters is stored in advance, so that when the correction is performed for the imaging of the fisheye camera of this model for the first time, the correspondence table is directly called to acquire the target virtual camera parameters and the target projection parameters according to the model of the fisheye camera.

[0093] S404, projecting the mapped set arc-shaped surface according to the target virtual camera parameters and the target projection parameters, so as to obtain a corrected target image.

[0094] It should be noted that the contents of steps S401, S402 and S404 are described in steps S101-S103.

[0095] Figure 5 is a flowchart of an image correction method according to yet another embodiment of the present application.

[0096] As shown in Figure 5 , the image correction method according to an embodiment of the present application comprises the following steps:

[0097] S501, obtaining an original image.

[0098] S502, applying the original image as a texture map to a set arc surface; wherein the planes on which the straight lines at both side edges of the set arc surface are parallel to the image plane on which the original image is located, and the intersection line between the set arc surface and the tangent plane perpendicular to the image plane is an arc.

[0099] S503, projecting the set arc surface after the texture map is applied according to a plurality of candidate virtual camera parameters and a plurality of candidate projection parameters, to obtain a reference image.

[0100] S504, in response to a user operation, determining a selected image from the reference images corresponding to each candidate virtual camera parameter and each candidate projection parameter.

[0101] S505, taking the candidate virtual camera parameter and the candidate projection parameter corresponding to the selected image as the corresponding target virtual camera parameter and the target projection parameter.

[0102] That is, the user can use a, b in different virtual camera parameters and fovy in the projection parameter to render the texture map of the set arc surface to the screen using OpenGL, that is, to obtain the corresponding reference image, and to determine whether the reference image is satisfactory. If one of the reference images is satisfactory, that is, as a selected image, the candidate virtual camera parameter and the candidate projection parameter corresponding to the selected image are taken as the corresponding target virtual camera parameter and the target projection parameter. If there is no image in the reference image that the user is satisfied with, the user can continue to project the original image using a, b in different virtual camera parameters and fovy in the projection parameter until a satisfactory image is projected. At this time, the candidate virtual camera parameter and the candidate projection parameter corresponding to the image are taken as the corresponding target virtual camera parameter and the target projection parameter.

[0103] S506, projecting the set arc surface after the texture map is applied according to the target virtual camera parameter and the target projection parameter, to obtain a corrected target image.

[0104] It should be noted that the contents of steps S501, S502 and S506 are described in steps S101-S103.

[0105] Thus, the fisheye video is directly mapped to a semicylindrical surface with the same pixel resolution, and the semicylindrical surface is rendered in the manner of opengl. Different rectification effects are obtained by modifying the direction of the virtual camera sight line, and different virtual camera sight line directions and virtual field of view angles are selected according to different scenes to achieve different rectification effects. In this way, the fisheye camera does not need to be calibrated in advance, and opengl is used to complete the rectification without relying on third-party library functions such as opencv; the video data can be processed in real time; and under the precondition of retaining a large field of view, objects close to the vehicle appear relatively normal.

[0106] In summary, according to the image correction method of the embodiment of the present application, the original image is first obtained and used as a texture map and attached to a set arc surface, wherein the planes in which the straight lines at the two side edges of the set arc surface are parallel to the image plane in which the original image is located, and the intersection line between the set arc surface and the tangent plane perpendicular to the image plane is arc-shaped, and then the set arc surface after the attachment is projected according to the target virtual camera parameters and the target projection parameters to obtain the corrected target image. Thus, the method does not need to calibrate the camera in advance, only needs to attach each frame of video content as a texture map to the set arc surface, and projects the set arc surface after the attachment according to the target virtual camera parameters and the target projection parameters to obtain the corrected target image, which can reduce the workload, improve the efficiency, and make the objects close to the vehicle appear relatively normal under the condition of retaining a large angle of view, thereby reducing the discomfort of the user when watching.

[0107] Figure 6 is a block schematic diagram of the image correction device according to the embodiment of the present application.

[0108] As shown in Figure 6 , the image correction device 600 according to the embodiment of the present application includes an acquisition module 601, an attachment module 602, and a first projection module 603.

[0109] The acquisition module 601 is configured to acquire an original image. The attachment module 602 is configured to attach the original image as a texture map to a set arc surface, wherein the planes in which the straight lines at the two side edges of the set arc surface are parallel to the image plane in which the original image is located, and the intersection line between the set arc surface and the tangent plane perpendicular to the image plane is arc-shaped. The first projection module 603 is configured to project the set arc surface after the attachment according to target virtual camera parameters and target projection parameters to obtain a corrected target image.

[0110] According to one embodiment of the present application, the image correction device further comprises a first determining module and a second determining module. The first determining module is configured to determine a radius of the visible region in the original image after the original image is obtained. The second determining module is configured to determine the radius of the set arc surface according to the radius of the visible region.

[0111] According to one embodiment of the present application, the mapping module 602 comprises a first determining unit, a second determining unit and a mapping unit. The first determining unit is configured to determine the origin of the three-dimensional coordinate system in which the set arc surface is located according to the center of the visible region in the original image. The X-axis in the three-dimensional coordinate system is parallel to the image plane, and the Y-axis is a straight line parallel to the set arc surface and perpendicular to the X-axis. The second determining unit is configured to determine the corresponding texture coordinates according to the X-axis and Y-axis coordinates of the target point on the set arc surface in the three-dimensional coordinate system. The mapping unit is configured to map the pixel unit with the texture coordinates in the texture coordinate system in which the original image is located to the target point according to the texture coordinates.

[0112] According to one embodiment of the present application, the second determining unit comprises a translation sub-unit and a processing sub-unit. The translation sub-unit is configured to translate the X-axis and Y-axis coordinates (X, Y) of the target point in the three-dimensional coordinate system according to the coordinates (x0, y0) of the center of the visible region in the texture coordinate system in which the original image is located. The processing sub-unit is configured to normalize the translated coordinates according to the resolution of the original image to obtain the texture coordinates corresponding to the target point.

[0113] According to one embodiment of the present application, the first projection module 603 comprises a third determining unit, a fourth determining unit and a first projection unit. The third determining unit is configured to determine the target optical axis direction of the virtual camera according to the direction parameter in the target virtual camera parameter. The fourth determining unit is configured to determine the target field of view angle of the virtual camera according to the angle parameter in the target projection parameter. The first projection unit is configured to project the mapped set arc surface to the virtual camera along the optical axis direction based on the target field of view angle to obtain the corrected target image.

[0114] According to one embodiment of the present application, the first projection unit comprises a first determining sub-unit, a second determining sub-unit and a projection sub-unit. The first determining sub-unit is configured to determine the virtual camera matrix according to the optical axis direction of the virtual camera. The second determining sub-unit is configured to determine the projection matrix between the imaging coordinate system of the virtual camera and the three-dimensional coordinate system in which the arc surface is located according to the target field of view angle of the virtual camera. The projection sub-unit is configured to project the mapped set arc surface to the virtual camera along the optical axis direction based on the virtual camera matrix and the projection matrix to obtain the corrected target image.

[0115] According to one of the embodiments of the present application, the image correction device described above further comprises a third determination module. The third determination module is configured to determine the target virtual camera parameter and the target projection parameter according to the model of the fisheye camera of the acquired original image.

[0116] According to one of the embodiments of the present application, the image correction device described above further comprises a second projection module, a fourth determination module and a setting module. The second projection module is further configured to project the set arc surface after the mapping according to the plurality of candidate virtual camera parameters and the plurality of candidate projection parameters to obtain the reference images. The fourth determination module is configured to determine the selected image from the reference images corresponding to each candidate virtual camera parameter and each candidate projection parameter in response to a user operation. The setting module is configured to take the candidate virtual camera parameter and the candidate projection parameter corresponding to the selected image as the corresponding target virtual camera parameter and target projection parameter respectively.

[0117] It should be noted that the details of the image correction device not disclosed in the embodiments of the present application can be found in the details disclosed in the image correction method of the embodiments of the present application, which will not be described here in detail.

[0118] According to the image correction device of the embodiments of the present application, the original image is acquired by the acquisition module, the original image is taken as a texture map and mapped to the set arc surface by the mapping module, the straight lines on the planes on both sides of the set arc surface are parallel to the image plane of the original image, and the intersection between the set arc surface and the tangent plane perpendicular to the image plane is an arc, and the first projection module projects the set arc surface after the mapping according to the target virtual camera parameter and the target projection parameter to obtain the corrected target image. Therefore, the device does not need to calibrate the camera in advance, only needs to take each frame of video content as a texture map and map it to the set arc surface, and projects the set arc surface after the mapping according to the target virtual camera parameter and the target projection parameter to obtain the corrected target image, which can reduce the workload, improve the efficiency, and make the display of the nearby objects relatively normal while preserving a large viewing angle, thereby reducing the discomfort of the user when watching.

[0119] Based on the above-mentioned embodiments, the present application further provides an electronic device.

[0120] The electronic device of the embodiments of the present application comprises a processor and a memory. The processor runs a program corresponding to the executable program code stored in the memory by reading the executable program code, so as to implement the image correction method described above.

[0121] The electronic device of the embodiment of the present application, by executing the image correction method, without calibrating the camera in advance, only needs to take each frame of video content as a texture map, paste on the set arc surface, and project the pasted set arc surface according to the target virtual camera parameter and the target projection parameter, so as to obtain the corrected target image, which can reduce the workload, improve the efficiency, and make the display of the nearby object relatively normal in the case of retaining a large viewing angle, and reduce the discomfort of the user when watching.

[0122] Based on the above embodiment, the present application further provides a non-transitory computer readable storage medium.

[0123] The non-transitory computer readable storage medium of the embodiment of the present application stores a computer program, and the computer program is executed by a processor to implement the above-mentioned image correction method.

[0124] The non-transitory computer readable storage medium of the embodiment of the present application, by executing the image correction method, without calibrating the camera in advance, only needs to take each frame of video content as a texture map, paste on the set arc surface, and project the pasted set arc surface according to the target virtual camera parameter and the target projection parameter, so as to obtain the corrected target image, which can reduce the workload, improve the efficiency, and make the display of the nearby object relatively normal in the case of retaining a large viewing angle, and reduce the discomfort of the user when watching.

[0125] Based on the above embodiment, the present application further provides a vehicle.

[0126] The vehicle of the embodiment of the present application comprises the above-mentioned electronic device.

[0127] The vehicle of the embodiment of the present application, when using the fisheye camera, without calibrating the fisheye camera in advance, only needs to take each frame of video content as a texture map, paste on the set arc surface, and project the pasted set arc surface according to the target virtual camera parameter and the target projection parameter, so as to obtain the corrected target image, which can reduce the workload, improve the efficiency, and make the display of the nearby object relatively normal in the case of retaining a large viewing angle, and reduce the discomfort of the user when watching.

[0128] Based on the above embodiment, the present application further provides a computer program product.

[0129] The computer program product of the embodiment of the present application, when the instruction processor in the computer program product is executed, executes the above-mentioned image correction method.

[0130] The computer program product of the embodiment of the present application, by executing the image correction method described above, does not need to calibrate the camera in advance, only needs to take each frame of video content as a texture map, paste it on the set arc surface, and project the pasted set arc surface according to the target virtual camera parameter and the target projection parameter, so that the corrected target image can be obtained, which can reduce the workload, improve the efficiency, and make the display of the nearby object relatively normal in the case of preserving a large viewing angle, and reduce the discomfort of the user when watching.

[0131] In the description of the present specification, the description referring to the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the present specification and the features of the different embodiments or examples without contradiction.

[0132] In addition, the terms "first", "second" are only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "a plurality of" is at least two, for example, two, three, etc., unless otherwise specifically limited.

[0133] Any process or method descriptions in flow charts or described elsewhere herein can be understood as representing code modules, segments, or portions of code that include one or more executable instructions for implementing specific logic functions (or steps) in the process, and that the scope of the preferred embodiments of the present application encompasses alterations, modifications, and variations of these code modules, segments, or portions of code that can be performed in an order different than the shown or discussed order, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those skilled in the art.

[0134] The logic and / or steps represented in the flowcharts and / or described herein, for example, can be considered as a sequence of instructions to implement logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, apparatus, or device, such as a computer-based system, processor- based system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For purposes of this specification, a "computer-readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer-readable medium can be a computer- readable storage medium or a computer-readable signal medium. The computer- readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium include the following: an electrical connection having one or more wires (electronic), a portable computer diskette (magnetic), a RAM (random access memory), a ROM (read-only memory), an EPROM (erasable programmable ROM), an EEPROM (electrically erasable programmable ROM), and a portable compact disc read-only memory (CD-ROM) (optical). In addition, the computer-readable medium can even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, for example, via the optical scanner of a device or device or via the acoustical scanning of the paper or other medium, then electronically captured, interpreted, or processed in a suitable manner if necessary, and then stored in a computer memory.

[0135] It should be understood that aspects of the application can be implemented in hardware, software, firmware or combinations thereof. In the above embodiments, various steps or methods can be implemented in software or firmware that is stored in memory and executed by a suitable instruction execution system. As such, in some embodiments, the steps or methods can be implemented in a combination of hardware and software. If implemented in hardware, as in another embodiment, any of the above techniques can be implemented with or without the use of a programmable data processing apparatus, using any of the following technologies: discrete logic circuits having logic gates for implementing logic functions on data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays, field programmable gate arrays, and the like.

[0136] Those of skill in the art would understand that information and signals can be represented using any of a variety of technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips that can be referenced throughout the above description can be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0137] In addition, each function unit in each embodiment of the present application can be integrated in one processing module, or each unit can be physically present separately, or two or more units can be integrated in one module. The integrated module can be realized in the form of hardware or in the form of a software function module. If the integrated module is realized in the form of a software function module and sold or used as an independent product, it can also be stored in a computer readable storage medium.

[0138] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application.

Claims

1. An image correction method characterized by, The method comprises the following steps: obtaining an original image; mapping the original image as a texture map to a set arc surface; wherein a straight line on the set arc surface at both side edges is parallel to an image plane on which the original image is located, and an intersection between the set arc surface and a tangent plane perpendicular to the image plane is an arc; projecting the mapped set arc surface according to target virtual camera parameters and target projection parameters to obtain a corrected target image, wherein the method comprises: determining a target optical axis direction of the virtual camera according to a direction parameter in the target virtual camera parameters; determining a target field of view angle of the virtual camera according to an angle parameter in the target projection parameters; projecting the mapped set arc surface to the virtual camera along the target optical axis direction based on the target field of view angle to obtain the corrected target image.

2. The method of claim 1, wherein, After the original image is obtained, the method further comprises: determining a radius of a visible region in the original image; determining a radius of the set arc surface according to the radius of the visible region.

3. The method of claim 1, wherein, The method of mapping the original image as a texture map to a set arc surface comprises: determining an origin of a three-dimensional coordinate system in which the set arc surface is located according to a center of a visible region in the original image; wherein an X-axis in the three-dimensional coordinate system is parallel to the image plane, and a Y-axis is parallel to a straight line coinciding with the set arc surface and perpendicular to the X-axis; determining corresponding texture coordinates according to X-axis and Y-axis coordinates of a target point on the set arc surface in the three-dimensional coordinate system; mapping a pixel unit having the texture coordinates in a texture coordinate system in which the original image is located to the target point according to the texture coordinates.

4. The method of claim 3, wherein, The method of determining corresponding texture coordinates according to X-axis and Y-axis coordinates of a target point on the set arc surface in the three-dimensional coordinate system comprises: performing translation on X-axis and Y-axis coordinates (X, Y) of the target point in the three-dimensional coordinate system according to coordinates (x0, y0) of a center of a visible region in a texture coordinate system in which the original image is located; performing normalization on the coordinates obtained after the translation according to a resolution of the original image to obtain the texture coordinates.

5. The method of claim 4, wherein, The method of projecting the mapped set arc surface to the virtual camera along the target optical axis direction based on the target field of view angle to obtain the corrected target image comprises: determining a virtual camera matrix according to the target optical axis direction of the virtual camera; determining a projection matrix between an imaging coordinate system of the virtual camera and a three-dimensional coordinate system in which the arc surface is located according to the target field of view angle of the virtual camera; projecting the mapped set arc surface to the virtual camera along the target optical axis direction based on the virtual camera matrix and the projection matrix to obtain the corrected target image.

6. The method according to any one of claims 1 to 4, characterized in that, Before the method of projecting the mapped set arc surface according to target virtual camera parameters and target projection parameters to obtain a corrected target image, the method further comprises: determining the target virtual camera parameters and the target projection parameters according to a model of a fisheye camera collecting the original image.

7. The method according to any one of claims 1 to 4, characterized in that, Before the projecting the mapped set arc-shaped surface according to the target virtual camera parameter and the target projection parameter to obtain the corrected target image, further comprising: projecting the mapped set arc-shaped surface according to a plurality of candidate virtual camera parameters and a plurality of candidate projection parameters to obtain reference images; in response to a user operation, determining a selected image from the reference images corresponding to each of the candidate virtual camera parameters and the candidate projection parameters; taking the candidate virtual camera parameter and the candidate projection parameter corresponding to the selected image as the corresponding target virtual camera parameter and the target projection parameter.

8. An image correction apparatus characterized by comprising: comprising: an acquisition module configured to acquire an original image; a mapping module configured to map the original image as a texture map to a set arc-shaped surface; wherein a straight line in the set arc-shaped surface at both side edges is parallel to an image plane on which the original image is located, and an intersection between the set arc-shaped surface and a tangent plane perpendicular to the image plane is an arc; a first projection module configured to project the mapped set arc-shaped surface according to a target virtual camera parameter and a target projection parameter to obtain a corrected target image, wherein the first projection module comprises: determining a target optical axis direction of the virtual camera according to a direction parameter in the target virtual camera parameter; determining a target field of view angle of the virtual camera according to an angle parameter in the target projection parameter; projecting the mapped set arc-shaped surface to the virtual camera along the target optical axis direction based on the target field of view angle to obtain the corrected target image.

9. An electronic device, comprising: comprising: a processor and a memory; wherein the processor runs a program corresponding to an executable program code stored in the memory by reading the executable program code, to implement the image correction method according to any one of claims 1-7.

10. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the image correction method according to any one of claims 1-7.

11. A vehicle characterized by comprising: comprising: the electronic device according to claim 9.

Images