Rolling shutter image correction method

Abstract

The present disclosure provides a rolling shutter image rectification method, by acquiring sparse three-dimensional point cloud and camera geometric parameters, firstly, the sparse three-dimensional point cloud is projected and the depth map is densified, and the dense depth map corresponding to the image to be rectified is generated, which overcomes the problem of holes and noise caused by the sparsity of point cloud; further, based on the dense depth map, the pose information of each row exposure time and the camera geometric parameters, the displacement mapping field between the target pixel position with the first row exposure time as the reference and the source pixel position in the image to be rectified is determined, the time difference of rolling shutter exposure and the influence of camera distortion are accurately modeled, and the problem that the pure visual method fails in the weak texture area is avoided; finally, the pixel value is sampled based on the displacement mapping field and the target pixel position is given, and the rectified image is generated, which eliminates the jelly effect distortion and makes the image geometric structure accurately recovered, providing high-quality input for downstream visual tasks.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to a method for correcting images from a rolling shutter. Background Technology

[0002] Rolling shutter image sensors are widely used due to their high dynamic range and low power consumption. However, their line-by-line exposure imaging mechanism results in a time difference between the exposure times of different lines of the image. When there is relatively high-speed movement between the camera and the scene being captured, this time difference can cause nonlinear geometric distortion in the acquired image, known as the "jelly effect," which severely affects the accuracy of subsequent visual algorithms.

[0003] Existing methods for removing the rolling shutter effect mainly fall into two categories: The first category is purely vision-based methods, which infer interline motion through image registration or optical flow estimation. These methods are highly dependent on the texture features of the scene, and often struggle to obtain accurate optical flow in areas with weak or repetitive textures, leading to correction failure. The second category is depth-assisted methods, which utilize RGB-D cameras to acquire scene geometric information and combine it with camera motion parameters for 3D reprojection correction. However, RGB-D cameras, such as structured light cameras or Time-of-Flight (ToF) cameras, are susceptible to ambient light interference. In outdoor bright light or distant scenes, their depth measurement quality degrades significantly, limiting their application scope.

[0004] In recent years, LiDAR (Light Detection and Ranging) has been widely used in autonomous driving and robotics perception due to its advantages of high measurement accuracy, long range, and insensitivity to lighting conditions. Although LiDAR can provide accurate 3D point clouds, its output is inherently sparse. The depth map generated by projecting sparse 3D point clouds onto the camera image plane often contains a large number of holes (i.e., areas without laser point projection) and outlier noise points caused by multipath effects or occlusion.

[0005] In summary, there is currently a lack of robust de-jelly effect technology that can simultaneously handle holes and noise interference in LiDAR projection depth maps and accurately couple camera distortion models. Summary of the Invention

[0006] To address the related technical problems in the background art, this disclosure provides a method for correcting rolling shutter images.

[0007] According to a first aspect of the present disclosure, a method for correcting images from a rolling shutter is provided, the method comprising: The method acquires an image to be corrected captured by a rolling shutter camera, a sparse three-dimensional point cloud corresponding to the image to be corrected, the pose information of the rolling shutter camera at the exposure time of each pixel row, and the camera geometric parameters of the rolling shutter camera, wherein the three-dimensional points in the sparse three-dimensional point cloud and the pixels in the image to be corrected have a spatial projection correspondence. Based on the sparse 3D point cloud and the camera geometric parameters, the sparse 3D point cloud is projected and the depth map is densed to generate a dense depth map that corresponds to the image to be corrected in the pixel dimension. Based on the dense depth map, the pose information, and the camera geometric parameters, a displacement mapping field is determined between the target pixel position and the source pixel position in the image to be corrected, wherein the target pixel position is the pixel position on the image plane based on the pose at the first exposure moment of the rolling shutter camera. Based on the displacement mapping field, pixel values ​​are sampled from the source pixel position in the image to be corrected, and the sampled pixel values ​​are assigned to the target pixel position to generate the corrected image.

[0008] As an optional embodiment, the step of projecting and densifying the sparse 3D point cloud based on the sparse 3D point cloud and the camera geometric parameters to generate a dense depth map corresponding to the image to be corrected in the pixel dimension includes: Based on the spatial distribution of the sparse 3D point cloud under the corresponding viewpoint of the image to be corrected, a depth map pyramid containing multiple resolution levels is constructed, and the level resolution corresponding to each resolution level is determined. Based on the resolution of each layer and the camera geometric parameters, the sparse 3D point cloud is processed to generate depth maps corresponding to each resolution layer. Based on the depth maps at each level, the final depth value corresponding to each pixel position in the image to be corrected is determined to obtain the dense depth map.

[0009] As an optional embodiment, the step of constructing a depth map pyramid containing multiple resolution levels based on the spatial distribution of the sparse 3D point cloud under the corresponding viewpoint of the image to be corrected, and determining the level resolution corresponding to each resolution level, includes: Obtain the depth value in the sparse 3D point cloud that is closest to the optical center of the rolling shutter camera, and determine this depth value as the closest depth value; Based on the nearest depth value and the camera geometric parameters, the layer resolution of the coarsest layer is determined; the layer resolution of the coarsest layer ensures that the pixel interval of the sparse 3D point cloud projected at the nearest depth value is less than one pixel. Based on the resolution of the coarsest level, the resolution of higher resolution levels is determined by doubling the resolution level by level, so as to obtain the resolution corresponding to each resolution level. The depth map pyramid is constructed based on the determined resolution of all levels.

[0010] As an optional embodiment, the step of determining the layer resolution of higher resolution layers based on the layer resolution of the coarsest layer, in a manner that increases with each resolution layer, to obtain the layer resolution corresponding to each resolution layer, includes: The resolution of the coarsest level is determined as the current level resolution; The current layer resolution is multiplied by a preset factor to obtain the layer resolution of the next higher resolution layer. Determine whether the resolution of the next higher resolution layer reaches a preset ratio of the resolution of the image to be corrected; In response to the fact that the resolution of the next higher resolution layer does not reach the preset ratio of the resolution of the image to be corrected, the next higher resolution layer is taken as the new current layer, and the step of multiplying the resolution of the current layer by a preset multiple is returned to be executed. In response to the next higher resolution layer reaching a preset ratio of the resolution of the image to be corrected, the determination of the higher resolution layer is stopped, and the next higher resolution layer is used as the highest resolution layer.

[0011] As an optional embodiment, the step of processing the sparse 3D point cloud according to each of the layer resolutions and the camera geometric parameters to generate depth maps corresponding to each of the resolution layers includes: For any resolution level, based on the resolution level and the camera geometric parameters, the sparse 3D point cloud is projected onto the image plane corresponding to the resolution level to generate the original depth map corresponding to the resolution level. The original depth map is filtered to remove unreliable depth values, thereby generating a layer depth map corresponding to the resolution level.

[0012] As an optional embodiment, the step of filtering the original depth map to remove unreliable depth values ​​and generating a layered depth map corresponding to the resolution level includes: The original depth map is subjected to noise removal processing to remove isolated noise points in the original depth map; For each pixel position in the depth map after noise removal processing, determine the number of 3D points in the sparse 3D point cloud projected to the pixel position. In response to the fact that the number of three-dimensional points in the sparse three-dimensional point cloud projected to the pixel location is one, the depth value of the three-dimensional point is determined as the depth value of the pixel location; In response to the fact that there are multiple 3D points in the sparse 3D point cloud projected to the pixel location, the depth value distribution of the multiple 3D points is statistically analyzed, and the depth value corresponding to the minimum peak value in the depth value distribution is selected as the depth value of the pixel location. Based on the depth values ​​determined at each pixel location, a layer depth map corresponding to the resolution layer is generated.

[0013] As an optional embodiment, determining the final depth value corresponding to each pixel position in the image to be corrected based on the depth maps of each level, in order to obtain the dense depth map, includes: For any pixel position in the image to be corrected, the depth maps of the depth map pyramid are traversed in order of progressively increasing resolution. It is then determined whether there is a valid depth value at the current resolution level and whether there is a valid depth value at the next higher resolution level. The valid depth value is the depth value obtained after the 3D point in the sparse 3D point cloud is projected onto the corresponding pixel position. In response to the existence of a valid depth value at the current resolution level and the absence of a valid depth value at the next higher resolution level, the current resolution level is determined as the target level, and the depth value of the pixel position in the target level is determined as the final depth value of the pixel position. In response to the fact that no valid depth value exists at any resolution level of the depth map pyramid, the depth value of the pixel position in the resolution level with the lowest resolution is determined as the final depth value of that pixel position. The dense depth map is generated based on the final depth value of each pixel location.

[0014] As an optional embodiment, determining the displacement mapping field between the target pixel position and the source pixel position in the image to be corrected based on the dense depth map, the pose information, and the camera geometric parameters includes: For any of the target pixel locations, back projection processing is performed on the target pixel location according to the camera geometric parameters, and combined with the depth value corresponding to the target pixel location in the dense depth map, to obtain the three-dimensional spatial point corresponding to the target pixel location; Using the first row of the rolling shutter camera as the current row, the three-dimensional spatial points are projected onto the image plane of the image to be corrected based on the pose information corresponding to the current row and the camera geometric parameters, to obtain the candidate pixel positions in the image to be corrected. Determine whether the row number of the candidate pixel position matches the row number of the current row; In response to the row number of the candidate pixel position matching the row number of the current row, the source pixel position corresponding to the target pixel position is determined based on the candidate pixel position; In response to a mismatch between the row number of the candidate pixel position and the row number of the current row, the current row is updated with the row number of the candidate pixel position, and the step of projecting the three-dimensional spatial point onto the image plane of the image to be corrected based on the pose information corresponding to the current row and the camera geometric parameters is iteratively executed until the row number of the candidate pixel position matches the row number of the current row or a preset termination condition is met. The displacement mapping field is generated based on the source pixel positions corresponding to each of the target pixel positions.

[0015] As an optional embodiment, the preset termination condition includes at least one of the following: the number of iterations reaches a preset threshold and the candidate pixel position exceeds the pixel coordinate boundary of the image to be corrected.

[0016] As an optional embodiment, the step of performing back-projection processing on the target pixel location based on the camera geometric parameters for any of the target pixel locations, and combining the depth value corresponding to the target pixel location in the dense depth map to obtain the three-dimensional spatial point corresponding to the target pixel location, includes: For any of the target pixel positions, the coordinates of the target pixel position are subjected to distortion correction processing based on the distortion parameters in the camera geometry parameters, so as to transform the coordinates of the target pixel position to obtain the distorted pixel position coordinates; Based on the intrinsic parameters in the camera geometry parameters and the depth value, the distortion-free pixel position coordinates are back-projected to three-dimensional spatial coordinates to obtain the three-dimensional spatial point corresponding to the target pixel position.

[0017] As an optional embodiment, the step of using the first row of the rolling shutter camera as the current row, and projecting the three-dimensional spatial points onto the image plane of the image to be corrected based on the pose information corresponding to the current row and the camera geometric parameters to obtain the candidate pixel positions in the image to be corrected includes: Using the first row of the rolling shutter camera as the current row, the three-dimensional spatial coordinates are transformed into three-dimensional coordinates in the camera coordinate system based on the pose information corresponding to the current row. Based on the intrinsic parameters in the camera geometry parameters, the three-dimensional coordinates in the camera coordinate system are projected into two-dimensional coordinates on the image plane of the image to be corrected, thereby obtaining the projected pixel positions; Based on the distortion parameters in the camera geometry parameters, the two-dimensional coordinates are transformed into distorted two-dimensional coordinates, which are then used as the candidate pixel positions.

[0018] As an optional embodiment, the step of sampling pixel values ​​from the source pixel positions in the image to be corrected based on the displacement mapping field, and assigning the sampled pixel values ​​to the target pixel positions to generate the corrected image, includes: The displacement mapping field is meshed and smoothed to generate a smooth displacement field; Based on the smooth displacement field, determine the source pixel position corresponding to each of the target pixel positions in the image to be corrected; The corrected image is generated by sampling pixel values ​​from the source pixel positions in the image to be corrected and assigning the sampled pixel values ​​to the corresponding target pixel positions.

[0019] In summary, by acquiring sparse 3D point clouds, pose information of the rolling shutter camera at each row of exposures, and camera geometric parameters, the sparse 3D point cloud is first projected and its depth map is densified to generate a dense depth map corresponding to the image to be corrected in the pixel dimension. This overcomes the problems of depth map holes and outlier noise points caused by the sparsity of LiDAR point clouds, ensuring that each pixel position has a reliable and effective depth value. Then, based on this dense depth map, pose information at each row of exposures, and camera geometric parameters, the displacement mapping field between the target pixel position (based on the pose at the first row of exposures) and the source pixel position in the image to be corrected is determined, thus modeling the rolling shutter's sequential... The actual pose change of each pixel row during the line exposure process and the influence of camera distortion avoid the problem of poor performance of pure vision methods in weak texture areas. At the same time, compared with RGB-D camera methods, it is not limited by ambient light and distance. Finally, based on this displacement mapping field, pixel values ​​are sampled from the image to be corrected and assigned to the target pixel position to generate the corrected image. Pixels from different rows taken at different times are unified to the same time and rearranged, eliminating rolling shutter distortion. The corrected image is equivalent to the image taken at the first line exposure time using global shutter mode. The geometric structure of objects in the image is accurately restored, providing high-quality input data for downstream vision tasks.

[0020] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0021] The accompanying drawings, which form part of this specification, illustrate embodiments of this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0022] This disclosure will become clearer with reference to the accompanying drawings and the following detailed description, wherein: Figure 1 This is a schematic diagram illustrating the jelly effect according to an embodiment of the present disclosure.

[0023] Figure 2 This is a schematic flowchart of a rolling shutter image correction method according to an embodiment of the present disclosure.

[0024] Figure 3 This is a schematic diagram of the depth map corresponding to the image to be corrected and the sparse 3D point cloud in an embodiment of this disclosure.

[0025] Figure 4 This is a schematic flowchart illustrating a rolling shutter image correction method according to another embodiment of the present disclosure.

[0026] Figure 5 This is a schematic diagram of a high-resolution depth map and a low-resolution depth map according to an embodiment of this disclosure.

[0027] Figure 6 This is a schematic diagram of a dense depth map according to an embodiment of the present disclosure.

[0028] Figure 7 This is a schematic flowchart illustrating a rolling shutter image correction method according to yet another embodiment of the present disclosure.

[0029] Figure 8 This is a timing diagram of the exposure of a rolling shutter camera according to an embodiment of this disclosure.

[0030] Figure 9 This is a schematic diagram of the displacement mapping field according to an embodiment of the present disclosure.

[0031] Figure 10 This is a schematic flowchart of a rolling shutter image correction method according to another embodiment of the present disclosure.

[0032] Figure 11 This is a schematic diagram comparing the image to be corrected and the image recovered based on the displacement mapping field, according to an embodiment of this disclosure.

[0033] Figure 12 This is a comparative diagram of the image to be corrected and the corrected image according to an embodiment of this disclosure.

[0034] Figure 13 This is a structural block diagram of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0035] Rolling shutter image sensors use a line-by-line exposure imaging mechanism, where different rows of pixels correspond to different exposure times. When there is relative motion between the camera and the scene, this difference in exposure times between rows can cause geometric distortion in the acquired image, known as the "rolling shutter effect."

[0036] like Figure 1 As shown, Figure 1 Figure A shows a checkerboard image taken by a rolling shutter camera in a horizontal motion state. Figure 1 Figure B in the image shows a checkerboard pattern image taken with a global shutter camera in the same scene. Figure 1 As can be seen in Figure A, the edges of the checkerboard pattern, which should be straight, appear as slanted or bent shapes in the image. Figure 1 In Figure B, the edges of the chessboard grid retain their original straight-line shape. (Through...) Figure 1 A comparison between Figure A and Figure B shows that the jelly effect in rolling shutter images significantly disrupts the true geometric structure of objects in the image.

[0037] To eliminate the rolling shutter effect, existing technologies mainly fall into two categories. One category is based on purely visual methods, inferring interline motion through image registration or optical flow estimation. However, these methods are highly dependent on scene texture features and struggle to obtain accurate results in areas with weak texture. The other category utilizes depth sensors to acquire scene geometric information and combines it with camera motion parameters for 3D reprojection correction. However, RGB-D cameras experience significant depth quality degradation in outdoor or distant scenes. While LiDAR can provide high-precision point clouds, the depth map formed after its projection onto the image plane contains numerous holes and outliers, making it difficult to directly use for rolling shutter effect correction.

[0038] In view of this, the present disclosure provides a rolling shutter image correction method to suppress depth data defects and achieve high-precision image correction. The technical solution of the present disclosure embodiments will be described in detail below with reference to the accompanying drawings.

[0039] Figure 2 This is a schematic flowchart illustrating a rolling shutter image correction method according to an embodiment of this disclosure. Figure 2 As shown, the rolling shutter image correction method of this disclosure embodiment may include the following steps: Step 110: Obtain the image to be corrected captured by the rolling shutter camera, the sparse 3D point cloud corresponding to the image to be corrected, the pose information of the rolling shutter camera at the exposure time of each pixel row, and the camera geometric parameters of the rolling shutter camera, wherein the 3D points in the sparse 3D point cloud and the pixels in the image to be corrected have a spatial projection correspondence.

[0040] In this embodiment of the disclosure, the image to be corrected can be an original RGB image with rolling shutter distortion acquired by a rolling shutter camera (or a rolling shutter image sensor) in motion (see [link to original document]). Figure 3 (See Figure C). Rolling shutter cameras use a line-by-line exposure imaging mechanism. There is a time difference in the exposure time of different pixel lines. When there is relative motion between the camera and the scene, the acquired image will produce geometric deformation.

[0041] In this embodiment of the disclosure, sparse 3D point clouds (see...) Figure 3 The D-map shown can be acquired using LiDAR or other depth sensing devices. Each 3D point in the sparse 3D point cloud can be projected onto the image plane of the image to be corrected using camera geometry parameters, falling on the corresponding pixel location or its neighborhood. That is, the 3D points in the sparse 3D point cloud and the pixels in the image to be corrected have a spatial projection correspondence.

[0042] Furthermore, pose information refers to the camera pose (including position and orientation) corresponding to the exposure time of each pixel row during the line-by-line exposure process of the rolling shutter camera. According to the imaging principle of the rolling shutter, each row on the sensor is exposed synchronously, but different rows do not begin exposure simultaneously. For a rolling shutter camera at a given moment, its readout time (i.e., the difference between the start time of the last row's exposure and the start time of the first row's exposure) is known. Assuming the readout time difference between each row is consistent, the pose at the start of each row's exposure can be estimated based on the row number. In this embodiment, pose information can be obtained by interpolating the timestamps of each row onto the IMU (Inertial Measurement Unit) data, or it can be obtained through visual-inertial odometry (VIO) or other sensor fusion methods that provide camera pose. The specific method of obtaining pose information is not limited in this embodiment.

[0043] Camera geometric parameters include, but are not limited to, intrinsic parameters (such as focal length and principal point coordinates) and extrinsic parameters (such as the rotation matrix and translation vector between the camera and LiDAR) of a rolling shutter camera, as well as distortion parameters (such as radial distortion coefficients and tangential distortion coefficients). These parameters describe the camera's imaging model and are the basis for realizing mutual projection transformation between 3D point clouds and 2D images.

[0044] In this step, multiple data sources, including the image to be corrected, sparse 3D point cloud, pose information, and camera geometric parameters, need to be aligned in both time and space. For example, the sparse 3D point cloud and the image to be corrected are projected to correspond using camera geometric parameters, and the pose information is aligned with the exposure time of each pixel row through hardware synchronization (such as hard synchronization) or timestamp interpolation. This step provides a complete data foundation for subsequent depth map densification, displacement mapping field calculation, and image correction.

[0045] Step 120: Based on the sparse 3D point cloud and camera geometric parameters, project and densify the sparse 3D point cloud to generate a dense depth map that corresponds to the image to be corrected in the pixel dimension.

[0046] Specifically, the sparse 3D point cloud is first projected onto the image plane of the image to be corrected according to the camera's geometric parameters (intrinsic, extrinsic, and distortion parameters) to obtain an initial sparse depth map. Due to the sparsity of the sparse 3D point cloud, the depth map generated by directly projecting it onto the camera's image plane (i.e., the image plane of the image to be corrected) often contains a large number of holes and outlier noise points caused by multipath effects or occlusion. In addition, the same pixel location may fall into multiple point clouds of different depth levels, resulting in the problem of foreground and background depth interleaving. For example, visible and invisible point clouds are interleaved, but in the color image (i.e., the image to be corrected), generally only the visible objects in the foreground can be observed, and invisible objects will not be imaged due to occlusion.

[0047] Therefore, this step densifies the projected depth data. One approach is to construct a multi-resolution depth map pyramid to address the inability to simultaneously capture distant details and dense nearby depth. In this pyramid, pixels at greater depths use higher-resolution depth levels, while pixels at smaller depths (closer distances) use lower-resolution levels. This ensures that the entire depth map retains more detail while maintaining dense depth. The pyramid's resolution levels can be dynamically adjusted based on the depth range of the depth map. For example, after determining the resolution of the coarsest level, intermediate levels are calculated exponentially by a preset number (e.g., 2) until a preset multiple or ratio of the original resolution is reached.

[0048] During the generation of each depth map layer, certain morphological and statistical operations can be performed to mitigate the problems caused by noise points. Specifically, an erosion operation can be performed first, for example, using a 3×3 rectangular window, to remove isolated and unreliable depth pixels caused by outlier noise points. Then, for each pixel position in the depth map after the erosion operation, the depth value distribution of all projection points at that pixel position is statistically analyzed, and the depth value corresponding to the peak with the smallest peak value in the depth value distribution is selected as the final depth value for that pixel position, in order to remove the depth values ​​corresponding to occluded background points. Since color images can only record the visible surface of the foreground scene, if the depth values ​​of occluded backgrounds are retained, it will cause the depth map and the color map to not be strictly aligned in space, thus affecting the accuracy of subsequent displacement mapping calculations. The above-mentioned erosion and statistical operations in this embodiment are executed independently at each resolution level. For each pixel location, the depth value of the highest resolution layer from all depth maps that guarantees no holes is selected as the final depth value for that pixel location. If any depth map layer of a pixel can cover more than one point, and the next higher resolution layer cannot cover more than one point, then that layer is determined to be the highest resolution layer (i.e., the layer with the richest detail) that can achieve dense, hole-free depth, and its depth is determined as the final depth. By selecting the most suitable depth value for each pyramid layer pixel by pixel, the final depth map can be obtained. At this point, the depth map is dense and single-layered (without overlapping depth values).

[0049] The depth value of each pixel in the dense depth map represents the distance of the scene point corresponding to that pixel location relative to the optical center of the rolling shutter camera. The unit of the depth value is, for example, meters, and it can be represented as a floating-point number, with 0 indicating an invalid value. After processing in step 120, the original sparse 3D point cloud is transformed into a dense depth map aligned with the pixels of the image to be corrected, providing an accurate geometric reference for subsequent 3D reprojection and displacement mapping calculations.

[0050] Step 130: Based on the dense depth map, pose information and camera geometric parameters, determine the displacement mapping field between the target pixel position and the source pixel position in the image to be corrected, wherein the target pixel position is the pixel position on the image plane based on the pose at the first exposure moment of the rolling shutter camera.

[0051] In this embodiment, the image space corresponding to the target pixel position is a pixel grid of a virtual global shutter image formed when all pixel rows of the rolling shutter camera are exposed simultaneously at the first exposure time. This virtual global shutter image has the same resolution as the image to be corrected in spatial dimension, but each pixel position corresponds to the camera pose at the same moment (the first exposure time). Specifically, the first-line pose P0 is the camera pose taken at the start of the first exposure time plus 0.5 times the exposure duration (i.e., the midpoint of the exposure).

[0052] The displacement mapping field is used to record which source pixel position in the image to be used to obtain the color value for each target pixel position in the aforementioned virtual global shutter image. That is, for the global shutter image at the first exposure moment, each pixel should come from the corresponding pixel in the rolling shutter image.

[0053] Specifically, for each target pixel location, firstly, based on the corresponding depth value in the dense depth map, the target pixel location is back-projected into three-dimensional space using camera geometric parameters to obtain the corresponding three-dimensional spatial point; then, based on the pose information of each exposure moment of the rolling shutter camera, the three-dimensional spatial point is reprojected onto the image plane of the image to be corrected, and the source pixel location that matches the target pixel location is determined through iterative search.

[0054] The specific process of iterative search is as follows: Let the pixel coordinates under the first row pose P0 be u0, and project them to the corresponding time pose P according to the row number. i The next pixel coordinate is u i If the row number matches the row number on the original image, then the actual pose at the time of exposure has been found. Therefore, this pixel should be mapped from the final projected pixel u. i Take the color value from the top. If projected onto P... i After u i If the line number and the line number of u0 are inconsistent, then continue to follow the line number of u0. i The row number corresponds to the pose P. i Continue projecting until the row number matches u0. The precision for determining row number consistency is 0.5 pixels in absolute error. If the projection fails to converge after, for example, the maximum number of iterations (10), the pixel is marked as invalid. If the projection occurs outside the image boundary during the projection process, it can also be marked as invalid.

[0055] In this embodiment, the construction of the displacement mapping field fully considers the exposure time difference between each row of the rolling shutter camera and the camera's distortion model. For example, distortion removal is performed during back projection, and distortion addition is performed during reprojection, which can accurately describe the pixel offset relationship introduced by the rolling shutter effect.

[0056] Step 140: Based on the displacement mapping field, sample pixel values ​​from the source pixel positions in the image to be corrected, and assign the sampled pixel values ​​to the target pixel positions to generate the corrected image.

[0057] Specifically, based on the determined displacement mapping field, each target pixel position is traversed: for each target pixel position, its corresponding source pixel position is found in the displacement mapping field, and then a color value is sampled from the source pixel position in the image to be corrected, and the sampled color value is assigned to the target pixel position. When the source pixel position is a sub-pixel coordinate (floating-point number), bilinear interpolation (i.e., linear interpolation) can be used for sampling.

[0058] For target pixel locations marked as invalid in the displacement mapping field, such as those where the valid source pixel location cannot be determined due to projection exceeding the image boundary or iteration non-convergence, no color fill is applied. Simultaneously, a corresponding invalid pixel mask is generated for subsequent use. For example, valid pixels in this mask are marked with a first preset value (e.g., 1), and invalid pixels are marked with a second preset value (e.g., 0), instructing downstream tasks to ignore the regions corresponding to invalid pixels.

[0059] After this step, the rolling shutter distortion caused by line-by-line exposure is effectively eliminated, and the corrected output image is equivalent to an image taken with global shutter mode at the first line exposure, thus restoring the geometric structure of objects in the image.

[0060] Figure 4 This is a flowchart illustrating another embodiment of the rolling shutter image correction method disclosed herein. Figure 4 As shown, the rolling shutter image correction method of this disclosure embodiment may further include the following steps: Step 121: Based on the spatial distribution of sparse 3D point cloud in the corresponding viewpoint of the image to be corrected, construct a depth map pyramid containing multiple resolution levels, and determine the level resolution corresponding to each resolution level.

[0061] As one implementation of this disclosure, this step can be specifically implemented as follows: First, the depth value closest to the optical center of the rolling shutter camera in the sparse 3D point cloud can be obtained, and this depth value can be determined as the nearest depth value.

[0062] In this embodiment, "nearest" can be relative to the camera's optical center, i.e., the minimum distance from all 3D points in the point cloud to the camera's optical center. In this embodiment, all 3D points in the input sparse 3D point cloud can be traversed, the distance from each 3D point to the optical center of the rolling shutter camera can be calculated, and the minimum value can be taken. This minimum value is the distance to the object closest to the sensor in the point cloud, ensuring that even at the coarsest level, nearby objects will not have voids.

[0063] Then, based on the nearest depth value and camera geometry parameters, the layer resolution of the coarsest layer is determined such that the pixel interval of the sparse 3D point cloud projected at the nearest depth value is less than one pixel.

[0064] Specifically, based on the nearest depth value and camera geometric parameters, such as the focal length parameter in the camera intrinsics, the maximum pixel interval between adjacent points after the point cloud is projected onto the image plane at that nearest depth value can be calculated. The resolution is then set to a value that makes this maximum pixel interval less than 1 pixel. This resolution is the resolution of the bottom layer (coarsest layer) of the pyramid.

[0065] As an alternative embodiment, when determining the resolution of the coarsest level, in addition to calculating based on the nearest depth value and camera focal length, the resolution of the coarsest level can also be determined adaptively based on the overall density distribution of the point cloud, or the resolution of the coarsest level can be inferred from a preset depth map quality requirement (such as the maximum allowable hole rate).

[0066] Secondly, based on the resolution of the coarsest level, the resolution of higher resolution levels is determined by doubling the resolution level by level, so as to obtain the resolution corresponding to each resolution level.

[0067] As one implementation, the resolution of layer 1 can be equal to the resolution of layer 0 multiplied by 2, the resolution of layer 2 can be equal to the resolution of layer 1 multiplied by 2, and so on, increasing in exponential multiples of 2.

[0068] Finally, a depth map pyramid is constructed based on all determined layer resolutions. This pyramid records the total number of layers and the specific width and resolution of each layer, i.e., a list of layer resolution specifications.

[0069] Figure 5 This is a schematic diagram comparing high-resolution depth maps and low-resolution depth maps provided in embodiments of this disclosure. Figure 5 As shown, the left image (E) is a high-resolution layered depth map, rich in detail but containing many void areas; the right image (F) is a low-resolution layered depth map, dense and void-free but with significant loss of detail. By constructing a multi-resolution depth map pyramid, we can balance the different needs of near objects requiring high resolution to ensure density and distant objects requiring low resolution to retain detail, so that the final depth map retains as much detail as possible while ensuring density and void-free conditions.

[0070] In a more specific implementation, based on the resolution of the coarsest level, the resolution of higher resolution levels is determined by progressively doubling the resolution at each level, thus obtaining the corresponding resolution for each resolution level. Specifically, this may include: determining the resolution of the coarsest level as the current resolution; multiplying the current resolution by a preset factor to obtain the resolution of the next higher resolution level; the preset factor can be 2x, or other factors such as 1.5x, 3x, etc.; further determining whether the resolution of the next higher resolution level reaches a preset proportion of the resolution of the image to be corrected. The preset ratio can be 1 / 4 of the original image resolution, or other ratios such as 1 / 2 or 1 / 8, depending on the accuracy requirements of the depth map in the actual application scenario. If the resolution of the next higher-resolution layer does not reach the preset ratio of the image resolution to be corrected, the next higher-resolution layer is taken as the new current layer, and the process of multiplying the resolution of the current layer by a preset factor is resumed. Conversely, if the resolution of the next higher-resolution layer reaches the preset ratio of the image resolution to be corrected, the determination of the resolution of higher-resolution layers stops, and this next higher-resolution layer is taken as the highest resolution layer. This process continues upwards, adding layers until the resolution of a certain layer reaches the preset ratio, at which point construction stops; this layer is the highest level of detail in the pyramid.

[0071] For example, assuming the nearest depth is 1 meter, the camera focal length is f, and the average pixel spacing at that depth is d, then the resolution of the coarsest layer needs to satisfy the condition that the pixel spacing between adjacent point clouds projected onto the image plane is less than 1 pixel at that resolution. If the calculated resolution of the coarsest layer is 1 / 16 of the original image resolution, then the first layer is 1 / 8 of the original image resolution, the second layer is 1 / 4 of the original image resolution, and so on, until it reaches 1 / 4 of the original image resolution.

[0072] In this step, by constructing a multi-resolution depth map pyramid, we can take into account the different needs of nearby objects requiring high resolution to ensure density and distant objects requiring low resolution to retain details. This allows the final depth map to retain as much detail as possible while ensuring density and eliminating voids.

[0073] Step 122: Based on the resolution of each layer and the camera geometric parameters, process the sparse 3D point cloud to generate depth maps corresponding to each resolution layer.

[0074] In this embodiment of the disclosure, the processing of this step can be performed independently for each layer in the pyramid, without affecting each other.

[0075] Specifically, for any resolution level, based on the resolution level and camera geometric parameters, the sparse 3D point cloud is projected onto the image plane corresponding to that resolution level to generate the original depth map corresponding to that resolution level. That is, using camera intrinsic parameters and distortion model, the original 3D point cloud is projected onto the image plane corresponding to that resolution to generate the original sparse depth map of that layer.

[0076] In this embodiment of the disclosure, since the sparse 3D point cloud itself is sparse, the original depth map obtained by direct projection contains a large number of holes (pixel positions without projection points) and outlier noise points caused by multipath effect or occlusion. It is necessary to filter the original depth map to remove unreliable depth values, including isolated outliers caused by sensor noise and depth values ​​corresponding to background points due to foreground and background occlusion.

[0077] Therefore, the original depth map is further filtered to remove unreliable depth values ​​and generate a layer depth map corresponding to the resolution layer.

[0078] In a more specific implementation, the original depth map is filtered to remove unreliable depth values, generating a layered depth map corresponding to the resolution level. This can be implemented as follows: First, noise removal is performed on the original depth map to remove isolated noise points.

[0079] The noise removal process can be implemented through morphological erosion, such as using a 3×3 rectangular window to remove isolated and unreliable depth pixels caused by outlier noise; outlier removal, such as judging and removing isolated outliers based on the number of valid pixels in each pixel's neighborhood; statistical filtering, such as removing points that deviate too far from the mean based on the statistical characteristics of depth values ​​in the neighborhood (e.g., mean and standard deviation); radius filtering, such as judging based on the number of valid points in the neighborhood; median filtering (replacing the center pixel value with the neighborhood median to suppress salt-and-pepper noise); or bilateral filtering (preserving edge features while removing noise). In this embodiment, the above operations can be performed independently at each resolution level.

[0080] Then, for each pixel location in the depth map after noise removal, the number of 3D points in the sparse 3D point cloud projected to that pixel location is determined.

[0081] That is, iterate through every pixel of the depth map of that layer and count the number of 3D points that the projection falls into at each pixel position.

[0082] As one implementation, in response to the fact that there is only one 3D point in the sparse 3D point cloud projected to the pixel location, the depth value of the 3D point is determined as the depth value of the pixel location. That is, if there is only one projection point in the pixel, its depth value is directly taken.

[0083] As another implementation, in response to the presence of multiple 3D points in the sparse 3D point cloud projected to the pixel location, the depth value distribution of these multiple 3D points is statistically analyzed, and the depth value corresponding to the minimum peak in the depth value distribution is selected as the depth value for that pixel location. Specifically, this can be achieved by statistically analyzing the histogram of these depth value distributions, identifying the peak, and selecting the peak with the minimum depth as the final depth value for that pixel location. When the same pixel location falls into multiple depth values, it can be confirmed that the foreground and background are intertwined. Since the color image can only capture the nearest foreground surface, the occluded background depth must be discarded to ensure strict alignment between the depth map and the color image.

[0084] As alternative implementation methods, in depth value distribution statistics, in addition to statistical histograms, methods such as kernel density estimation can also be used to estimate the distribution of depth values ​​and determine the peak value.

[0085] Finally, based on the depth values ​​determined at each pixel location, a layer depth map corresponding to that resolution layer is generated. The above processing is performed independently at each resolution layer.

[0086] To continue with the example, suppose at a certain resolution level, a pixel location is affected by three LiDAR point projections with depth values ​​of 2.1m, 2.3m, and 5.8m. After noise removal, the distribution of these three depth values ​​is statistically analyzed, and the peak value is found in the 2.1-2.3m range (the minimum peak value). The depth value corresponding to this range (such as the average of 2.1m or 2.3m) is selected as the final depth value for that pixel location, while the occluded background depth value of 5.8m is discarded.

[0087] In this processing step, noise removal and pixel-by-pixel depth value determination can effectively remove isolated outliers caused by sensor noise, while removing the depth values ​​corresponding to occluded background points, so that the depth maps at each level are strictly aligned with the color map in space while maintaining density.

[0088] Step 123: Based on the depth maps of each level, determine the final depth value corresponding to each pixel position in the image to be corrected, so as to obtain a dense depth map.

[0089] Specifically, for any pixel position in the image to be corrected, the depth maps of each level of the depth map pyramid are traversed in order of progressively increasing resolution, and it is determined in turn whether there is a valid depth value at the current resolution level and whether there is a valid depth value at the next higher resolution level.

[0090] The effective depth value is the depth value obtained by projecting a 3D point in a sparse 3D point cloud onto the corresponding pixel position. In this embodiment, the criteria for determining whether there is an effective depth value at the current resolution level and whether there is an effective depth value at the next higher resolution level can be, for example, whether the pixel position falls into ≥1 point cloud projection in the depth map of that layer, i.e., whether the pixel is effectively non-zero.

[0091] As one implementation, in response to the existence of a valid depth value at the current resolution level and the absence of a valid depth value at the next higher resolution level, the current resolution level is determined as the target level, and the depth value of the pixel position in the target level is determined as the final depth value of the pixel position.

[0092] In this way, this step can find a critical layer, such as "the current layer has a valid depth value, and the next layer (higher resolution) does not have a valid depth value". This critical layer is the highest resolution layer at which the pixel can be guaranteed to be hole-free.

[0093] As an alternative implementation, in response to the absence of valid depth values ​​at any resolution level of the depth map pyramid, the depth value of the pixel location in the lowest resolution level is determined as the final depth value of that pixel location.

[0094] Specifically, since the resolution of the coarsest layer is designed so that the pixel interval of the projection of the sparse 3D point cloud at the nearest depth value is less than one pixel, it can ensure that there are no holes in the entire map. Even if a certain pixel is invalid in all higher layers, it will eventually be able to obtain the depth value of the coarsest layer, ensuring that the final output depth map is absolutely hole-free.

[0095] Finally, a dense depth map is generated based on the final depth value of each pixel location.

[0096] All selected depth values ​​are then stitched together to form a final dense, single-layer floating-point depth map with the same resolution as the image to be corrected. The unit of depth value is, for example, meters, and can be represented by floating-point numbers, such as 0 indicating an invalid value. There are no restrictions on the farthest and nearest distances.

[0097] Figure 6 This is a schematic diagram of a dense depth map provided in an embodiment of this disclosure. Figure 6 As shown, after pixel-by-pixel optimal layer selection and fusion, the final depth map maintains both its dense, hole-free characteristics (meaning all pixel locations have effective depth values) and maximizes the preservation of scene detail, exhibiting clear structure in areas of depth discontinuity and object edges. This depth map is single-layered, with no overlapping depth values, and corresponds one-to-one with the image to be corrected in the pixel dimension.

[0098] In some other embodiments of this disclosure, when traversing each layer pixel by pixel for judgment, in addition to traversing from the coarsest layer to the highest resolution layer, more efficient search strategies such as binary search can be used to accelerate the localization of the target layer. After determining the target layer, in addition to directly taking the depth value of that layer, the depth values ​​of adjacent layers can be weighted and fused to further improve the accuracy and continuity of the depth values.

[0099] To illustrate further, suppose the depth map pyramid has four layers (layer 0 is the coarsest layer, and layer 3 is the highest resolution layer). For a pixel location in the image to be corrected, layer 0 has a valid depth value, layer 1 also has a valid depth value, layer 2 has a valid depth value, but layer 3 does not. Then, following the traversal rules: if layer 0 has a valid depth value and layer 1 also has one → continue; if layer 1 has a valid depth value and layer 2 also has one → continue; if layer 2 has a valid depth value and layer 3 does not have one → the condition is met, layer 2 is determined as the target layer, and the depth value of that pixel location in layer 2 is taken as the final depth value.

[0100] In this step, by performing pixel-by-pixel fusion and optimization among the depth maps of each level, the highest-resolution depth value without holes can be selected for each pixel location. This ensures that the entire depth map is dense and hole-free while preserving the scene's detail information to the maximum extent. At the same time, due to the fallback protection mechanism of the coarsest level, it is ensured that all pixel locations obtain a valid depth value, and the final output depth map is dense and single-layered, with no overlapping depth values ​​between layers.

[0101] Figure 7 This is a schematic flowchart illustrating another embodiment of the rolling shutter image correction method of this disclosure. Figure 7 As shown, the rolling shutter image correction method of this disclosure embodiment may further include the following steps: Step 131: For any target pixel location, perform back projection processing on the target pixel location according to the camera geometric parameters, and combine it with the depth value corresponding to the target pixel location in the dense depth map to obtain the three-dimensional spatial point corresponding to the target pixel location.

[0102] One implementation method involves performing distortion correction processing on the coordinates of any target pixel location based on distortion parameters in the camera's geometric parameters. This transforms the target pixel location coordinates into distorted pixel coordinates. Then, based on the intrinsic parameters and depth value in the camera's geometric parameters, the distorted pixel coordinates are back-projected onto three-dimensional space coordinates to obtain the three-dimensional point corresponding to the target pixel location. In this embodiment, distortion correction processing allows the distorted pixel coordinates to be substituted into the inverse function of the distortion model to calculate ideal distortion-free normalized coordinates. Then, using camera intrinsic parameters (focal length, principal point coordinates, etc.), the distorted pixel coordinates are combined with the depth value to recover the coordinates of the three-dimensional point in the camera coordinate system. The calculation method for the three-dimensional point coordinates is, for example, X = (u - cx). Z / fx, Y = (v - cy) Z / fy, where (u, v) are the pixel coordinates after distortion correction, Z is the depth value, (fx, fy) is the focal length, and (cx, cy) are the principal point coordinates.

[0103] like Figure 8 As shown, each row on the rolling shutter image sensor is exposed synchronously, but different rows do not begin exposure at the same time. The first row begins exposure first, followed by each subsequent row with a delayed start, and the last row begins exposure last. Figure 8 This illustrates the source of the pose information corresponding to the current row. Each row has its own exposure time and pose, which are used in subsequent iterations to update the current row with the row number of the candidate pixel position. In other words, the projection is re-switched to the exposure time and pose corresponding to that row.

[0104] In distortion correction, besides substituting pixel coordinates into the inverse function of the distortion model for calculation, an iterative approximation method can be used to solve for the distorted coordinates, or a look-up table (LUT) can be used to pre-store the distortion mapping relationship to accelerate processing. In the backprojection process, in addition to using the backprojection formula for the pinhole camera model, other camera models (such as fisheye camera models, F-theta models, etc.) can also be used, depending on the type of camera and distortion model employed.

[0105] To illustrate further, suppose the target pixel location coordinates are (320, 240), corresponding to a depth of 5.2 meters, with camera intrinsic parameters fx=800, fy=800, cx=640, cy=360, and distortion parameters k1 and k2. First, the (320, 240) pixel is distorted according to the distortion parameters, resulting in distorted pixel coordinates (322, 238). Then, the 3D spatial coordinates are calculated using the back projection formula: X = (322-640)5.2 / 800 ≈ -2.067 meters, Y = (238-360)5.2 / 800 ≈ -0.793 meters, Z = 5.2 meters.

[0106] By performing distortion correction on the target pixel location before backprojection, the impact of camera lens distortion on the accuracy of 3D point reconstruction can be eliminated, resulting in more accurate 3D point coordinates. Simultaneously, by combining the depth value of each pixel location in the dense depth map, a corresponding 3D point can be reconstructed for each target pixel location, providing an accurate geometric reference for subsequent reprojection and displacement mapping calculations.

[0107] Step 132: Using the first row of the rolling shutter camera as the current row, project the three-dimensional spatial points onto the image plane of the image to be corrected based on the pose information corresponding to the current row and the camera geometric parameters, to obtain the candidate pixel positions in the image to be corrected.

[0108] As one implementation method, firstly, the first row of a rolling shutter camera can be used as the current row. Based on the pose information corresponding to the current row, the 3D spatial coordinates are transformed into 3D coordinates in the camera coordinate system. This transformation can be achieved by using extrinsic parameters (rotation matrix and translation vector) from the world coordinate system to the camera coordinate system based on the pose information of the current row. Then, based on the intrinsic parameters in the camera geometry, the 3D coordinates in the camera coordinate system are projected into 2D coordinates on the image plane of the image to be corrected, obtaining the projected pixel positions. Specifically, intrinsic parameters in the camera geometry (such as focal length, principal point coordinates, etc.) can be used to project the 3D points in the camera coordinate system into ideal, distortion-free pixel coordinates. The projection formula is, for example, u = fx. X / Z + cx, v = fy Y / Z + cy, where (X, Y, Z) are the three-dimensional coordinates in the camera coordinate system. Finally, based on the distortion parameters in the camera geometry, the two-dimensional coordinates are transformed into distorted two-dimensional coordinates to serve as candidate pixel positions. In this embodiment, the distortion processing, for example, involves substituting the ideal, undistorted pixel coordinates into a positive function of the distortion model to calculate the distorted true pixel coordinates.

[0109] In coordinate transformation, besides using rotation matrices and translation vectors, pose can also be represented and transformed using quaternions or Euler angles. During projection, in addition to the projection formula for the pinhole camera model, projection formulas for other camera models can also be used, depending on the type of camera employed. Furthermore, in distortion processing, besides substituting ideal coordinates into the positive function of the distortion model for calculation, a lookup table can be used to pre-store distortion mapping relationships to accelerate processing.

[0110] Assume the coordinates of a 3D point are -2.067, -0.793, 5.2. The pose information corresponding to the current row is the rotation matrix R and the translation vector t. The camera intrinsic parameters are fx=800, fy=800, cx=640, cy=360, and the distortion parameters are k1 and k2. First, transform the 3D point from the world coordinate system to the camera coordinate system based on R and t to obtain the 3D coordinates (1.5, 0.8, 5.2) in the camera coordinate system. Then, calculate the ideal pixel coordinates using the projection formula: u = 800. 1.5 / 5.2 + 640 ≈ 870.77, v = 800 0.8 / 5.2 + 360 ≈ 483.08; Finally, distortion processing is performed on (870.77, 483.08) according to the distortion parameters to obtain the candidate pixel position with distortion (872.3, 484.7).

[0111] By projecting a 3D spatial point onto the image plane of the image to be corrected based on the pose information corresponding to the current line, the imaging position of that 3D spatial point by a rolling shutter camera at the current line exposure time can be simulated. By simultaneously considering camera intrinsic parameters and distortion parameters during the projection process, the projected candidate pixel positions are more accurate and can truly reflect the actual pixel coordinates recorded by the rolling shutter camera at the current line exposure time.

[0112] Step 133: Determine whether the row number of the candidate pixel position matches the row number of the current row.

[0113] Step 134: In response to the row number of the candidate pixel position matching the row number of the current row, determine the source pixel position corresponding to the target pixel position based on the candidate pixel position.

[0114] Step 135: In response to the mismatch between the row number of the candidate pixel position and the row number of the current row, update the current row with the row number of the candidate pixel position, and iteratively execute the step of projecting the 3D spatial points onto the image plane of the image to be corrected based on the pose information corresponding to the current row and the camera geometric parameters, until the row number of the candidate pixel position matches the row number of the current row or a preset termination condition is met. That is, after updating the current row with the row number of the candidate pixel position, iteratively execute steps 131 to 133.

[0115] The preset termination condition may include at least one of the following: the number of iterations reaches a preset threshold and the candidate pixel position exceeds the pixel coordinate boundary of the image to be corrected.

[0116] Specifically, in steps 133 to 135 mentioned above, during the initial projection, the first line of the rolling shutter camera (e.g., line number r=0, which can be referenced) is used. Figure 8 Using row 0 as the current row, the pose information P0 corresponding to this row is used to project the 3D spatial point onto the image plane of the image to be corrected, obtaining the candidate pixel position u_new. Then, it is determined whether the row number of the candidate pixel position u_new is consistent with the current row number r. If they are consistent, the correct source pixel position has been found, and the mapping relationship is that the target pixel position u0 points to the candidate pixel position u_new. If they are inconsistent, the current row number r is updated with the row number of the candidate pixel position u_new, that is, r = the row number of u_new. The 3D spatial point is then re-projected onto the image plane of the image to be corrected using the pose information P_r corresponding to this row, obtaining a new candidate pixel position. This process is repeated iteratively until the row number of the candidate pixel position is consistent with the row number of the current row. If the iteration count reaches a preset threshold (e.g., 10 times) and convergence is not achieved, or if the candidate pixel position exceeds the pixel coordinate boundary of the image to be corrected, the iteration is stopped and the target pixel position is marked as invalid.

[0117] As an optional embodiment, the precision for determining whether the row number of a candidate pixel position matches the current row number can be within an absolute error of 0.5 pixels, that is, the absolute value of the difference between the row number of the candidate pixel position and the current row number is less than 0.5 rows. For example, assuming the current row number r=0 and the row number of the candidate pixel position u_new is 0.3, then |0.3-0|=0.3<0.5, indicating a row number match; if the row number of the candidate pixel position u_new is 0.8, then |0.8-0|=0.8>0.5, indicating a row number mismatch. A 0.5-pixel threshold is used because image row numbers are integer indices, while the coordinates of the candidate pixel position obtained from projection are floating-point numbers, resulting in calculation errors. The 0.5-pixel threshold corresponds to the middle position between two adjacent rows; that is, when the row number of the candidate pixel position falls within half a row of pixels of the current row, the point is considered to belong to the current row. This effectively avoids judgment errors caused by floating-point calculation errors. If the threshold is set too small, floating-point errors may cause row numbers that should match to be misjudged as mismatched, increasing unnecessary iterations. If it is set too large, pixels belonging to adjacent rows may be misjudged as belonging to the current row, resulting in decreased mapping accuracy. Therefore, 0.5 pixels in this embodiment is a reasonable threshold that balances robustness and accuracy.

[0118] In this embodiment, the iterative execution process involves the following: for the same three-dimensional spatial point on an object, it is unknown which row in the rolling shutter camera it was exposed on. It is necessary to try projecting different row poses to find the row number that matches the actual exposure time of the three-dimensional point. When the row number used for projection equals the row number of the projected pixel, it can be determined that the three-dimensional spatial point was recorded at that pixel position at that actual shooting time. This pixel position is the correct position to extract color from the original image to be corrected.

[0119] As other implementations, besides updating row numbers sequentially from the first row for projection during iterative execution, more efficient search strategies such as binary search can be used to accelerate the row number matching process. When determining whether row numbers match, in addition to using an absolute error precision of 0.5 pixels, the judgment threshold can be adjusted according to the precision requirements of the actual application scenario. Preset termination conditions, besides reaching a preset threshold for the number of iterations and the candidate pixel position exceeding the image boundary, can also include other conditions, such as the deviation between the depth value corresponding to the candidate pixel position and the expected depth value exceeding a certain range.

[0120] Assume the target pixel position u0 is located in the first row (row number 0). The 3D point X, projected under pose P0, yields a candidate pixel position u_new located in row 5. Since row number 5 of u_new is inconsistent with the current row number 0, the current row is updated with row number 5, and the 3D point X is reprojected using pose P5, resulting in a new candidate pixel position u_new located in row 3. The current row is then updated again with row number 3, and reprojected using pose P3, yielding another candidate pixel position u_new located in row 3. At this point, the row number 3 used for projection matches the row number 3 of the projected pixel. The iteration stops, and the source pixel position is determined as the final projected candidate pixel position.

[0121] By employing an iterative search method, the source pixel position corresponding to the target pixel position can be accurately located, fully considering the exposure time difference between rows of a rolling shutter camera. Simultaneously, by setting preset termination conditions (maximum number of iterations and image boundary judgment), abnormal situations such as non-convergence or exceeding boundaries can be effectively handled, ensuring the robustness of the algorithm. Pixels that do not converge or exceed boundaries are marked as invalid; subsequent mesh smoothing can tolerate a small number of invalid points without affecting the overall correction effect.

[0122] Step 136: Generate a displacement mapping field based on the source pixel positions corresponding to each target pixel position.

[0123] Specifically, for each target pixel location, after processing in steps 131 to 135, there is a corresponding source pixel location (or it is marked as invalid). The mapping relationships between all target pixel locations and their corresponding source pixel locations are summarized to obtain the displacement mapping field. This displacement mapping field records which pixel location in the image to be corrected should be used to obtain the color value for each pixel location in the target image (a virtual global shutter image based on the pose at the first exposure time). Each pixel location in the displacement mapping field records a displacement vector (dx, dy), representing the coordinate offset between the target pixel location and the source pixel location. Target pixel locations marked as invalid are also marked in the displacement mapping field for subsequent processing.

[0124] Figure 9 This is a schematic diagram of the displacement mapping field provided in an embodiment of this disclosure. Figure 9 As shown, the displacement mapping field visually represents the pixel mapping relationship from the target image to the image to be corrected. Different colors or arrow directions indicate different displacement directions and magnitudes. Through this displacement mapping field, the pixel offset pattern introduced by the jelly effect can be observed intuitively; that is, in the direction of motion, the pixel offset of different rows is different, forming a continuously changing displacement field.

[0125] When generating the displacement mapping field, in addition to recording the source pixel position corresponding to each target pixel position, the confidence level or weight information corresponding to each target pixel position can also be recorded to reflect the reliability of the mapping relationship. For pixel positions marked as invalid, in addition to direct marking, they can also be filled by neighborhood interpolation and other methods to improve the integrity of the displacement mapping field and the subsequent correction effect.

[0126] To further illustrate, assuming the target image resolution is 1920×1080, after processing steps 131 to 135, the source pixel position corresponding to the target pixel position (100, 200) is (98, 205). Then, the displacement vector recorded at position (100, 200) in the displacement mapping field is... This indicates that the target pixel position should obtain its color value from position (98, 205) in the image to be corrected. After traversing all target pixel positions, the complete displacement mapping field can be obtained.

[0127] By generating a displacement mapping field, the pixel offset relationship caused by the exposure time difference between each row of the rolling shutter is accurately recorded in data form, providing a direct pixel mapping basis for subsequent image correction. This displacement mapping field can accurately describe the pixel offset pattern introduced by the rolling shutter effect, including the offset difference between different rows and the influence of the distortion model.

[0128] Figure 10This is a schematic flowchart illustrating another embodiment of the rolling shutter image correction method disclosed herein. Figure 10 As shown, the rolling shutter image correction method of this disclosure embodiment may further include the following steps: Step 141: Perform meshing and smoothing on the displacement mapping field to generate a smooth displacement field.

[0129] One approach is to first construct a uniformly distributed regular grid on the target image plane. This regular grid comprises multiple grid vertices and grid cells formed by adjacent vertices. The target image plane is an image plane based on the pose of the rolling shutter camera at the first exposure moment. For example, the grid size can be balanced between precision and data sparsity, such as using a uniform 20×20 pixel grid.

[0130] Then, for any grid vertex, the neighborhood range corresponding to that grid vertex in the displacement mapping field is determined, and the pixel displacement vector of the valid pixels within the neighborhood range is obtained; where the valid pixels are the pixels in the displacement mapping field that are not marked as invalid, and the displacement vector is used to indicate the coordinate offset between the target pixel position and the source pixel position in the image to be corrected.

[0131] For example, the neighborhood range can be 3×3 grid units, that is, the neighborhood range formed by extending one grid unit in the horizontal and one in the vertical directions with any grid vertex as the center. It includes the area covered by the grid vertex itself and its surrounding adjacent grid vertices, covering a total range of 9 grid vertices (3 rows × 3 columns) or 8 adjacent grid vertices (excluding itself).

[0132] For example, suppose the vertex coordinates of a regular mesh are Let , where i represents the horizontal index and j represents the vertical index. The neighborhood range is the 3×3 grid vertex region covered by the horizontal index from i-1 to i+1 and the vertical index from j-1 to j+1. Taking grid vertex (5, 5) as an example, its 3×3 grid unit neighborhood range includes vertices (4,4), (4, 5), (4, 6), (5, 4), (5, 5), (5, 6), (6, 4), (6, 5), and (6, 6), that is, the vertex itself and its eight adjacent grid vertices in the top, bottom, left, right, and diagonal directions. Within this neighborhood range, the displacement vectors of all valid pixels (i.e., the displacement vectors corresponding to the pixel positions that are not marked as invalid) are collected to calculate the smooth displacement vector of the grid vertex. In this embodiment, when the neighborhood is too small (e.g., 1×1 grid units), it only includes the grid vertex itself, which cannot effectively suppress the influence of local noise and non-smoothness on the displacement field. Conversely, when the neighborhood is too large (e.g., 5×5 grid units or larger), it may introduce too many distant displacement vectors, causing the smoothed displacement field to lose local details, and significantly increasing the computational load. The 3×3 grid units in this embodiment represent an empirical value that strikes a reasonable balance between smoothing effect and detail preservation. It effectively suppresses local outliers in the displacement field while retaining relatively fine structural information, and is therefore the preferred implementation.

[0133] In alternative solutions, the neighborhood extent can also use 5×5 grid units or adaptively adjust its size based on depth variations. Smaller neighborhood extents are used in areas with drastic depth changes to preserve detail, while larger neighborhood extents are used in areas with gentle depth changes to achieve better smoothing. Furthermore, the neighborhood shape is not limited to rectangles; it can also be circular or rhomboid, among other shapes.

[0134] Secondly, a weighted average is calculated on the pixel displacement vectors of the effective pixels within the neighborhood to obtain the smooth displacement vector corresponding to the grid vertex. Optionally, a Gaussian weighting function can be used for the weighted average, with pixel displacement vectors closer to the vertex having a larger weight.

[0135] Subsequently, a sparse grid mapping field is constructed based on the smooth displacement vectors corresponding to multiple grid vertices. Optionally, this sparse grid mapping field may only contain the grid vertex coordinates and their corresponding smooth displacement vectors, with a data volume far less than the number of pixels in the entire image.

[0136] Finally, for any pixel location on the target image plane, interpolation is performed based on the displacement vector of the grid cell to which the pixel location belongs in the sparse grid mapping field to obtain the smooth displacement vector of the pixel location, so as to generate a smooth displacement field.

[0137] Furthermore, for any pixel location on the target image plane, the grid cell to which the pixel location belongs in the sparse grid mapping field can be determined; the smoothed displacement vectors corresponding to the grid vertices of the grid cell to which the pixel location belongs are obtained; then, based on the relative position of the pixel location within the grid cell, bilinear interpolation is performed on the obtained smoothed displacement vectors to obtain the smoothed displacement vector corresponding to the pixel location; based on the smoothed displacement vectors corresponding to each pixel location, a smoothed displacement field is generated. This smoothed displacement field has the same resolution as the original image, and each pixel has a smoothed displacement vector. At this point, the smoothed displacement field satisfies the manifold assumption (one-to-one mapping), fundamentally eliminating the ghosting problem caused by depth jumps.

[0138] In this embodiment, regarding the grid division method, in addition to uniform division, adaptive division can also be performed based on image content (such as regions with depth variations), using a denser grid in regions with drastic depth variations to retain more details. In vertex smoothing, besides using Gaussian weights for weighted averaging, other weighting methods such as inverse distance weighting and bilateral weighting can also be used. Regarding the interpolation method for pixels within the grid, besides bilinear interpolation, other interpolation methods such as bicubic interpolation can also be used. This embodiment does not limit the above content.

[0139] For example, assuming the target image resolution is 1920×1080 and a uniform 20×20 pixel grid is used, the number of grid vertices is approximately (1920 / 20+1)×(1080 / 20+1)≈97×55. For a given grid vertex, the displacement vectors of all valid pixels within a 3×3 grid unit radius around that vertex are taken and weighted using Gaussian weights to obtain the smooth displacement vector of that vertex. For any pixel within the grid, based on its relative position within that grid unit, bilinear interpolation is performed on the smooth displacement vectors of the four vertices to obtain the smooth displacement vector of that pixel.

[0140] Thus, by using mesh smoothing, local non-smoothness issues in the displacement mapping field can be effectively eliminated, ensuring that the entire displacement field satisfies the manifold assumption and avoiding ghosting artifacts caused by multiple target pixel positions mapping to the same source pixel position due to depth jumps. Simultaneously, the intermediate representation of the sparse mesh mapping field reduces the amount of data and improves computational efficiency.

[0141] Step 142: Determine the source pixel position in the image to be corrected for each target pixel position based on the smoothed displacement field.

[0142] One implementation involves first iterating through the target pixel positions in the image to be corrected and obtaining the smooth displacement vector corresponding to each target pixel position from the smooth displacement field. This smooth displacement vector records the coordinate offset (dx, dy) of the source pixel position relative to the target pixel position in the image to be corrected. Then, based on the target pixel position and the smooth displacement vector, the source pixel position in the image to be corrected is calculated; specifically, for example, source pixel position coordinates = target pixel position coordinates + smooth displacement vector, i.e., (x_s, y_s) = (x_t + dx, y_t + dy). Finally, it is determined whether the source pixel position is within the pixel boundary of the image to be corrected. In this embodiment, the pixel boundary can be the effective pixel coordinate range of the image to be corrected, i.e., from 0 to W-1 in the width direction and from 0 to H-1 in the height direction. In response to the source pixel location being within the pixel boundary of the image to be corrected, the source pixel location is determined as a valid source pixel location. Correspondingly, in response to the source pixel location being outside the pixel boundary of the image to be corrected, the target pixel location is marked as invalid. The invalid target pixel location will not be assigned a color value in subsequent steps.

[0143] When calculating the source pixel position, in addition to directly adding the target pixel position coordinates to the smoothed displacement vector, the displacement vector can be further scaled or adjusted to adapt to different correction requirements. When determining whether the source pixel position is valid, in addition to checking whether it is located inside or outside the image boundary, other validity conditions such as whether the depth value corresponding to the source pixel position is within a reasonable range can also be checked.

[0144] Assuming the target pixel position is (100, 200) and the corresponding smoothed displacement vector is (-2, +5), the source pixel position is (98, 205). If the resolution of the image to be corrected is 1920×1080, then (98, 205) is within the pixel boundary (0≤98<1920, 0≤205<1080), therefore this source pixel position is determined as a valid source pixel position. If the target pixel position is (1910, 200) and the corresponding smoothed displacement vector is (+20, 0), then the source pixel position is (1930, 200), which exceeds the image width boundary, and this target pixel position is marked as invalid.

[0145] This step determines the source pixel location in the image to be corrected for each target pixel location and marks invalid pixels. Determining the valid source pixel locations provides precise sampling coordinates for subsequent color sampling, while marking invalid pixels ensures the reliability of the subsequent corrected image by not assigning color values ​​to invalid areas, thus avoiding erroneous color information.

[0146] Step 143: Sample pixel values ​​from the source pixel positions in the image to be corrected, and assign the sampled pixel values ​​to the corresponding target pixel positions to generate the corrected image.

[0147] In this embodiment of the disclosure, each target pixel position is traversed, and it is first determined whether the target pixel position is a valid pixel. A valid pixel is a target pixel position that has not been marked as invalid, such as a target pixel position whose source pixel position is located within the pixel boundary of the image to be corrected in step 142. That is, in step 142, a target pixel position whose source pixel position is located within the pixel boundary of the image to be corrected is determined as a valid source pixel position, and correspondingly, the target pixel position is a valid pixel.

[0148] As one implementation, in response to the target pixel location being a valid pixel, the source pixel location corresponding to the target pixel location in the image to be corrected is obtained. The source pixel location can be sub-pixel coordinates, such as floating-point numbers. Then, based on the source pixel location, a bilinear interpolation algorithm is used to sample the color value corresponding to the source pixel location from the image to be corrected, and the sampled color value is assigned to the target pixel location. In this step, bilinear interpolation uses the color values ​​of the four integer pixel locations surrounding the source pixel location, and performs a weighted average based on the relative distance between the source pixel location and the surrounding pixels to obtain the color value of the sub-pixel location.

[0149] As another implementation, in response to a target pixel location being an invalid pixel, no color value is assigned to the target pixel location; that is, invalid pixels are not filled with any color. Simultaneously, an invalid pixel mask is generated. In this mask, valid pixels are marked with a first preset value, and invalid pixels are marked with a second preset value, instructing downstream tasks to ignore the regions corresponding to invalid pixels. For example, valid pixels can be marked as 1 (or 255), and invalid pixels as 0. This invalid pixel mask is used by downstream tasks (such as 3D reconstruction) to ignore unreliable regions.

[0150] In the color sampling of this embodiment, in addition to bilinear interpolation, nearest neighbor interpolation can also be used, i.e., directly taking the color value of the nearest integer pixel position, or bicubic interpolation, i.e., using more surrounding pixels for higher precision interpolation, etc. For the handling of invalid pixels, in addition to not assigning a color value, they can also be filled with the color value of the neighboring valid pixels, or filled with the background color, etc.

[0151] For example, suppose the color value at the source pixel location (98.3, 205.7) in the image to be corrected needs to be sampled. The bilinear interpolation algorithm uses the color values ​​of four integer pixel locations around this location (98, 205), (99, 205), (98, 206), and (99, 206), and calculates a weighted average based on the distance of (98.3, 205.7) relative to these four pixels to obtain the color value of this sub-pixel location. This color value is then assigned to the target pixel location (100, 200). For target pixel locations marked as invalid, no color value is assigned, and their corresponding positions are marked as 0 in the invalid pixel mask.

[0152] After processing in step 143, the rolling shutter distortion caused by line-by-line exposure is effectively eliminated. The corrected output image is equivalent to an image captured using global shutter mode at the first line exposure, and the geometric structure of objects in the image is restored. Simultaneously, the invalid pixel mask provides downstream tasks with information indicating pixel reliability, enabling downstream algorithms to identify and ignore unreliable regions, thereby improving the overall system robustness.

[0153] See Figure 11 and Figure 12 . Figure 11 This is a schematic diagram comparing the original rolling shutter image and the directly recovered global shutter image provided in an embodiment of this disclosure. For example... Figure 11 As shown, the left image G is the original image from the rolling shutter, and the right image H is the global shutter image recovered by directly applying the displacement mapping field (i.e., without the meshing smoothing process in step 141). Figure 11 As can be seen from the H diagram on the right, since the displacement mapping field is not a smooth field, its direct application leads to defects such as ghosting in some local areas with varying depths. Figure 12 This is a schematic diagram comparing the pre-correction image and the post-correction image provided in an embodiment of this disclosure, such as... Figure 12 As shown, the left X-image is the image before correction (original image from the rolling shutter), and the right Y-image is the corrected image obtained after complete processing from steps 141 to 143. This corrected image is generated after gridding smoothing to generate a smooth displacement field, determining the source pixel positions corresponding to each target pixel position, sampling color values ​​from the image to be corrected, and assigning them to the target pixel positions. Through the gridding smoothing process in step 141, the displacement field satisfies the manifold assumption, eliminating the ghosting problem. Then, through color sampling and assignment in steps 142 and 143, the geometric structure of the object in the corrected image is accurately restored. Figure 11 and Figure 12 In comparison, the gridding smoothing method provided in this disclosure can effectively eliminate ghosting defects caused by directly applying displacement mapping fields and improve image correction quality.

[0154] In summary, by acquiring sparse 3D point clouds, pose information of the rolling shutter camera at each row of exposures, and camera geometric parameters, the sparse 3D point cloud is first projected and its depth map is densified to generate a dense depth map corresponding to the image to be corrected in the pixel dimension. This overcomes the problems of depth map holes and outlier noise points caused by the sparsity of LiDAR point clouds, ensuring that each pixel position has a reliable and effective depth value. Then, based on this dense depth map, pose information at each row of exposures, and camera geometric parameters, the displacement mapping field between the target pixel position (based on the pose at the first row of exposures) and the source pixel position in the image to be corrected is determined, thus modeling the rolling shutter's sequential... The actual pose change of each pixel row during the line exposure process and the influence of camera distortion avoid the problem of poor performance of pure vision methods in weak texture areas. At the same time, compared with RGB-D camera methods, it is not limited by ambient light and distance. Finally, based on this displacement mapping field, pixel values ​​are sampled from the image to be corrected and assigned to the target pixel position to generate the corrected image. Pixels from different rows taken at different times are unified to the same time and rearranged, eliminating rolling shutter distortion. The corrected image is equivalent to the image taken at the first line exposure time using global shutter mode. The geometric structure of objects in the image is accurately restored, providing high-quality input data for downstream vision tasks.

[0155] Below, for reference Figure 13 This describes an electronic device according to embodiments of the present disclosure. The electronic device may be either or both of a first device and a second device, or a standalone device independent of them, which may communicate with the first device and the second device to receive acquired input signals from them.

[0156] Figure 13 A block diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0157] like Figure 13 As shown, the electronic device includes one or more processors and memory.

[0158] A processor can be a central processing unit (CPU) or other form of processing unit with data processing and / or instruction execution capabilities, and can control other components in an electronic device to perform desired functions.

[0159] The memory can store one or more computer program products, and the memory can include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program products can be stored on the computer-readable storage medium, and the processor can run the computer program products to implement the rolling shutter image correction and / or other desired functions described in the various embodiments of this disclosure above.

[0160] In one example, the electronic device may also include input devices and output devices, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0161] In addition, the input device may also include, for example, a keyboard, a mouse, etc.

[0162] This output device can output various information to the outside, including determined distance information, direction information, etc. The output device may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0163] Of course, for the sake of simplicity, Figure 13 Only some of the components of the electronic device relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device may include any other suitable components depending on the specific application.

[0164] In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps in the rolling shutter image correction methods according to various embodiments of this disclosure as described in the foregoing portions of this specification.

[0165] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0166] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform the steps in the rolling shutter image correction method according to various embodiments of this disclosure as described in the foregoing portion of this specification.

[0167] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0168] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0169] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0170] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0171] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0172] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0173] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0174] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A method for correcting images from a rolling shutter, characterized in that, The method includes: The method acquires an image to be corrected captured by a rolling shutter camera, a sparse three-dimensional point cloud corresponding to the image to be corrected, the pose information of the rolling shutter camera at the exposure time of each pixel row, and the camera geometric parameters of the rolling shutter camera, wherein the three-dimensional points in the sparse three-dimensional point cloud and the pixels in the image to be corrected have a spatial projection correspondence. Based on the sparse 3D point cloud and the camera geometric parameters, the sparse 3D point cloud is projected and the depth map is densed to generate a dense depth map that corresponds to the image to be corrected in the pixel dimension. Based on the dense depth map, the pose information, and the camera geometric parameters, a displacement mapping field is determined between the target pixel position and the source pixel position in the image to be corrected, wherein the target pixel position is the pixel position on the image plane based on the pose at the first exposure moment of the rolling shutter camera. Based on the displacement mapping field, pixel values ​​are sampled from the source pixel position in the image to be corrected, and the sampled pixel values ​​are assigned to the target pixel position to generate the corrected image.

2. The method according to claim 1, characterized in that, The step of projecting and densifying the sparse 3D point cloud based on the sparse 3D point cloud and the camera's geometric parameters to generate a dense depth map corresponding to the image to be corrected in the pixel dimension includes: Based on the spatial distribution of the sparse 3D point cloud under the corresponding viewpoint of the image to be corrected, a depth map pyramid containing multiple resolution levels is constructed, and the level resolution corresponding to each resolution level is determined. Based on the resolution of each layer and the camera geometric parameters, the sparse 3D point cloud is processed to generate depth maps corresponding to each resolution layer. Based on the depth maps at each level, the final depth value corresponding to each pixel position in the image to be corrected is determined to obtain the dense depth map.

3. The method according to claim 2, characterized in that, The step of constructing a depth map pyramid containing multiple resolution levels based on the spatial distribution of the sparse 3D point cloud under the corresponding viewpoint of the image to be corrected, and determining the level resolution corresponding to each resolution level, includes: Obtain the depth value in the sparse 3D point cloud that is closest to the optical center of the rolling shutter camera, and determine this depth value as the closest depth value; Based on the nearest depth value and the camera geometric parameters, the layer resolution of the coarsest layer is determined; the layer resolution of the coarsest layer ensures that the pixel interval of the sparse 3D point cloud projected at the nearest depth value is less than one pixel. Based on the resolution of the coarsest level, the resolution of higher resolution levels is determined by doubling the resolution level by level, so as to obtain the resolution corresponding to each resolution level. The depth map pyramid is constructed based on the determined resolution of all levels.

4. The method according to claim 3, characterized in that, The process of determining the layer resolution of higher resolution layers based on the layer resolution of the coarsest layer, in a progressively doubling manner, to obtain the layer resolution corresponding to each resolution layer, includes: The resolution of the coarsest level is determined as the current level resolution; The current layer resolution is multiplied by a preset factor to obtain the layer resolution of the next higher resolution layer. Determine whether the resolution of the next higher resolution layer reaches a preset ratio of the resolution of the image to be corrected; In response to the fact that the resolution of the next higher resolution layer does not reach the preset ratio of the resolution of the image to be corrected, the next higher resolution layer is taken as the new current layer, and the step of multiplying the resolution of the current layer by a preset multiple is returned to be executed. In response to the next higher resolution layer reaching a preset ratio of the resolution of the image to be corrected, the determination of the higher resolution layer is stopped, and the next higher resolution layer is used as the highest resolution layer.

5. The method according to any one of claims 2 to 4, characterized in that, The step of processing the sparse 3D point cloud according to the resolution of each layer and the camera geometric parameters to generate depth maps corresponding to each resolution layer includes: For any resolution level, based on the resolution level and the camera geometric parameters, the sparse 3D point cloud is projected onto the image plane corresponding to the resolution level to generate the original depth map corresponding to the resolution level. The original depth map is filtered to remove unreliable depth values, thereby generating a layer depth map corresponding to the resolution level.

6. The method according to claim 5, characterized in that, The step of filtering the original depth map to remove unreliable depth values ​​and generating a layer depth map corresponding to the resolution level includes: The original depth map is subjected to noise removal processing to remove isolated noise points in the original depth map; For each pixel position in the depth map after noise removal processing, determine the number of 3D points in the sparse 3D point cloud projected to the pixel position. In response to the fact that the number of three-dimensional points in the sparse three-dimensional point cloud projected to the pixel location is one, the depth value of the three-dimensional point is determined as the depth value of the pixel location; In response to the fact that there are multiple 3D points in the sparse 3D point cloud projected to the pixel location, the depth value distribution of the multiple 3D points is statistically analyzed, and the depth value corresponding to the minimum peak value in the depth value distribution is selected as the depth value of the pixel location. Based on the depth values ​​determined at each pixel location, a layer depth map corresponding to the resolution layer is generated.

7. The method according to any one of claims 2 to 4, characterized in that, The step of determining the final depth value corresponding to each pixel position in the image to be corrected based on the depth maps at each level, in order to obtain the dense depth map, includes: For any pixel position in the image to be corrected, the depth maps of the depth map pyramid are traversed in order of progressively increasing resolution. It is then determined whether there is a valid depth value at the current resolution level and whether there is a valid depth value at the next higher resolution level. The valid depth value is the depth value obtained after the 3D point in the sparse 3D point cloud is projected onto the corresponding pixel position. In response to the existence of a valid depth value at the current resolution level and the absence of a valid depth value at the next higher resolution level, the current resolution level is determined as the target level, and the depth value of the pixel position in the target level is determined as the final depth value of the pixel position. In response to the fact that no valid depth value exists at any resolution level of the depth map pyramid, the depth value of the pixel position in the resolution level with the lowest resolution is determined as the final depth value of that pixel position. The dense depth map is generated based on the final depth value of each pixel location.

8. The method according to any one of claims 1 to 4, characterized in that, The step of determining the displacement mapping field between the target pixel position and the source pixel position in the image to be corrected, based on the dense depth map, the pose information, and the camera geometric parameters, includes: For any of the target pixel locations, back projection processing is performed on the target pixel location according to the camera geometric parameters, and combined with the depth value corresponding to the target pixel location in the dense depth map, to obtain the three-dimensional spatial point corresponding to the target pixel location; Using the first row of the rolling shutter camera as the current row, the three-dimensional spatial points are projected onto the image plane of the image to be corrected based on the pose information corresponding to the current row and the camera geometric parameters, to obtain the candidate pixel positions in the image to be corrected. Determine whether the row number of the candidate pixel position matches the row number of the current row; In response to the row number of the candidate pixel position matching the row number of the current row, the source pixel position corresponding to the target pixel position is determined based on the candidate pixel position; In response to a mismatch between the row number of the candidate pixel position and the row number of the current row, the current row is updated with the row number of the candidate pixel position, and the step of projecting the three-dimensional spatial point onto the image plane of the image to be corrected based on the pose information corresponding to the current row and the camera geometric parameters is iteratively executed until the row number of the candidate pixel position matches the row number of the current row or a preset termination condition is met. The displacement mapping field is generated based on the source pixel positions corresponding to each of the target pixel positions.

9. The method according to claim 8, characterized in that, The preset termination conditions include at least one of the following: the number of iterations reaches a preset threshold and the candidate pixel position exceeds the pixel coordinate boundary of the image to be corrected.

10. The method according to claim 8, characterized in that, For any of the target pixel locations, the back-projection process is performed on the target pixel location based on the camera geometric parameters, and combined with the depth value corresponding to the target pixel location in the dense depth map to obtain the three-dimensional spatial point corresponding to the target pixel location, including: For any of the target pixel positions, the coordinates of the target pixel position are subjected to distortion correction processing based on the distortion parameters in the camera geometry parameters, so as to transform the coordinates of the target pixel position to obtain the distorted pixel position coordinates; Based on the intrinsic parameters in the camera geometry parameters and the depth value, the distortion-free pixel position coordinates are back-projected to three-dimensional spatial coordinates to obtain the three-dimensional spatial point corresponding to the target pixel position.

11. The method according to claim 8, characterized in that, The step of using the first row of the rolling shutter camera as the current row, and projecting the three-dimensional spatial points onto the image plane of the image to be corrected based on the pose information corresponding to the current row and the camera's geometric parameters to obtain the candidate pixel positions in the image to be corrected, includes: Using the first row of the rolling shutter camera as the current row, the three-dimensional spatial coordinates are transformed into three-dimensional coordinates in the camera coordinate system based on the pose information corresponding to the current row. Based on the intrinsic parameters in the camera geometry parameters, the three-dimensional coordinates in the camera coordinate system are projected into two-dimensional coordinates on the image plane of the image to be corrected, thereby obtaining the projected pixel positions; Based on the distortion parameters in the camera geometry parameters, the two-dimensional coordinates are transformed into distorted two-dimensional coordinates, which are then used as the candidate pixel positions.

12. The method according to any one of claims 1 to 4, characterized in that, The step of sampling pixel values ​​from the source pixel positions in the image to be corrected based on the displacement mapping field, and assigning the sampled pixel values ​​to the target pixel positions to generate the corrected image includes: The displacement mapping field is meshed and smoothed to generate a smooth displacement field; Based on the smooth displacement field, determine the source pixel position corresponding to each of the target pixel positions in the image to be corrected; The corrected image is generated by sampling pixel values ​​from the source pixel positions in the image to be corrected and assigning the sampled pixel values ​​to the corresponding target pixel positions.

Images