A method, device, electronic equipment and storage medium for texture mapping

Abstract

The present disclosure provides a method and device for texture mapping, electronic equipment and storage medium, relates to the field of image processing, and particularly relates to the field of three-dimensional reconstruction technology. The specific implementation scheme is: obtaining a target pose of a row in which a projection point corresponding to a target ground patch in an image is located through an image first row pose of the image, determining the projection point of the target ground patch in the image based on a target ground patch space coordinate and a target pose corresponding to the target ground patch, and performing texture mapping on the target ground patch based on the projection point coordinate. According to the embodiment of the present disclosure, the target ground patch is projected based on the image pose of the row in which the projection point corresponding to the target ground patch is located, the accuracy of the obtained target ground patch corresponding pose is improved, the pixel drift caused by the rolling shutter exposure and the exposure delay is corrected, the projection point calculation accuracy is improved, the texture mapping accuracy is improved, and the map feature labeling accuracy is further improved.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to the field of three-dimensional reconstruction and other related technologies. Background Technology

[0002] 3D reconstruction has wide applications in various fields, such as building high-precision maps, reconstructing cultural relics, and reconstructing scenes. Texture mapping is an important step in 3D reconstruction, which refers to projecting the texture information of a 2D image onto the corresponding 3D scene or object in the fused point cloud data to obtain the texture map of that scene or object. Summary of the Invention

[0003] This disclosure provides a method, apparatus, electronic device, and storage medium for texture mapping to improve texture mapping accuracy.

[0004] According to one aspect of this disclosure, a method for texture mapping is provided, comprising:

[0005] The spatial coordinates of the target ground patch are obtained based on the fused point cloud data, wherein the ground patch is obtained by dividing the ground corresponding to the fused point cloud data based on a preset resolution;

[0006] Based on the spatial coordinates of the target ground patch, obtain an image including the target ground patch;

[0007] Obtain the first row pose of the image, wherein the first row pose is the pose of the camera when it captures the image;

[0008] Based on the first row pose of the image, the target pose of the target ground patch in the image is estimated to be the row where the projection point of the target ground patch is located in the image.

[0009] Based on the spatial coordinates of the target ground patch and the target pose, determine the target projection point of the target ground patch in the image;

[0010] Texture mapping is performed on the target ground patch based on the target projection point of the target ground patch in the image.

[0011] According to another aspect of this disclosure, an apparatus for texture mapping is provided, comprising:

[0012] The spatial coordinate acquisition module is used to acquire the spatial coordinates of the target ground patch based on the fused point cloud data, wherein the ground patch is obtained by dividing the ground corresponding to the fused point cloud data based on a preset resolution;

[0013] The image acquisition module is used to acquire an image including the target ground patch based on the spatial coordinates of the target ground patch;

[0014] The first-row pose acquisition module is used to acquire the first-row pose of the image, wherein the first-row pose is the pose of the camera when acquiring the image;

[0015] The target pose estimation module is used to estimate the target pose of the row in which the projection point of the target ground patch is located in the image based on the pose of the first row of the image.

[0016] The target projection point determination module is used to determine the target projection point of the target ground patch in the image based on the spatial coordinates of the target ground patch and the target pose;

[0017] The texture mapping module is used to perform texture mapping on the target ground patch based on the target projection point of the target ground patch in the image.

[0018] According to one aspect of this disclosure, an electronic device is provided, comprising:

[0019] At least one processor; and

[0020] A memory communicatively connected to the at least one processor; wherein,

[0021] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform any of the texture mapping methods described above.

[0022] According to one aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform any of the texture mapping methods described above.

[0023] According to one aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the texture mapping method described above.

[0024] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0025] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0026] Figure 1 This is a schematic diagram of a first embodiment of a texture mapping apparatus provided in this disclosure;

[0027] Figure 2This is a schematic diagram illustrating the method for calculating the coordinates of a target ground patch in the texture mapping method provided in this disclosure;

[0028] Figure 3a This is a flowchart illustrating a method for constructing a target point cloud height grid in the texture mapping method provided in this disclosure;

[0029] Figure 3b This is a schematic diagram of a triangular mesh;

[0030] Figure 4 This is a schematic diagram of the exposure mechanism of a rolling shutter camera;

[0031] Figure 5 This is a flowchart illustrating a method for obtaining the target pose corresponding to a target ground patch in the texture mapping method provided in this disclosure;

[0032] Figure 6 This is a schematic diagram of a framework for a texture mapping method provided in this disclosure;

[0033] Figure 7 This is a schematic diagram of a first embodiment of a texture mapping apparatus provided in this disclosure;

[0034] Figure 8 This is a block diagram of an electronic device used to implement the texture mapping method of the embodiments of this disclosure. Detailed Implementation

[0035] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0036] The process of 3D reconstruction typically involves fusing collected point cloud data of scenes and objects, then extracting texture information from the point cloud data to apply texture maps to the corresponding scenes or objects, thus obtaining the final reconstruction result. Taking the construction of a high-precision map as an example, the process usually involves fusing point cloud data collected for roads, extracting road surface features such as lane lines, traffic lights, and curbs from the point cloud data to create a road texture map, which is then applied to the fused road surface to obtain the high-precision map.

[0037] Extracting road surface features from point cloud data typically involves automated or manual annotation of these features using point cloud reflectance values. However, in certain situations or scenarios, point cloud reflectance values ​​are unstable, making it difficult to extract clear road surface features. For example, when lane lines are worn, the reflectance values ​​of the lane lines are very similar to those of the ground, making it difficult to extract clear lane line features. Images, on the other hand, can capture rich texture information, and even worn lane lines remain clearly visible in the image. Therefore, related technologies use texture mapping to project image texture information onto the road texture map corresponding to the fused point cloud data, thereby improving the annotation accuracy of high-precision map features.

[0038] Image acquisition typically utilizes either a global shutter camera or a rolling shutter camera. Because rolling shutter cameras acquire images by exposing the sensor line by line, the exposure time for pixels in different rows of the same image varies, resulting in different camera poses (referred to as image poses in this paper) for different rows of pixels within the same image. In related technologies, when performing texture mapping on images acquired using rolling shutter cameras, the image pose error caused by the rolling shutter exposure method is often ignored, thus propagating this error to the texture map. This leads to drift in map features within the texture map, resulting in poor texture mapping accuracy. For example, lane lines on the texture map may not align with the point cloud data, leading to inaccurate map feature labeling.

[0039] To improve texture mapping accuracy, this disclosure provides a method, apparatus, electronic device, and storage medium for texture mapping. The texture mapping method provided in this disclosure is first described by way of example:

[0040] The texture mapping method disclosed herein can be applied to any electronic device that has texture mapping. Such electronic devices can be servers, computers, mobile terminals, etc.

[0041] like Figure 1 As shown, Figure 1 This is a flowchart illustrating a first embodiment of the texture mapping method provided in this disclosure, which may specifically include the following steps:

[0042] Step S101: Obtain the spatial coordinates of the target ground patch based on the fused point cloud data, wherein the ground patch is obtained by dividing the ground corresponding to the fused point cloud data based on a preset resolution;

[0043] Step S102: Based on the spatial coordinates of the target ground patch, obtain an image including the target ground patch;

[0044] Step S103: Obtain the first row pose of the image, wherein the first row pose is the pose of the camera when it captures the image;

[0045] Step S104: Based on the first row pose of the image, estimate the target pose of the row where the projection point of the target ground patch is located in the image;

[0046] Step S105: Based on the spatial coordinates of the target ground patch and the target pose, determine the target projection point of the target ground patch in the image;

[0047] Step S106: Perform texture mapping on the target ground patch based on the target projection point of the target ground patch in the image.

[0048] By applying the embodiments of this disclosure, the first row pose of an image containing a target ground patch is obtained, the image pose of the row containing the projection point of the target ground patch is estimated based on the first row pose, and the target ground patch is projected based on the image pose of the row containing the projection point of the target ground patch. This takes into account the image pose error caused by line-by-line exposure of the rolling shutter, improves the accuracy of the obtained image pose of the target ground patch, corrects pixel drift caused by line-by-line exposure of the rolling shutter and exposure delay, improves the accuracy of projection point calculation, thereby improving the accuracy of texture mapping and thus improving the accuracy of map feature annotation.

[0049] The following is an exemplary description of steps S101-S106 above:

[0050] In step S101, the ground corresponding to the fused point cloud data can be divided according to a preset resolution to obtain various ground patches. As mentioned above, texture mapping refers to projecting the texture information of an image onto the 3D scene or object corresponding to the fused point cloud data to obtain a texture map of the scene or object. The aforementioned preset resolution can represent the ground area represented by one pixel in the texture map.

[0051] The preset resolution can be set according to actual needs. For example, the resolution can be 0.03125m, that is, each pixel in the texture map obtained after texture mapping of the ground represents a ground patch of size 0.03125m * 0.03125m. Since the area of ​​each ground patch is small, in this embodiment of the disclosure, each ground patch can be regarded as a point.

[0052] The aforementioned fused point cloud data can specifically be fused ground point cloud data. The ground corresponding to the fused point cloud data is the actual road corresponding to the fused ground point cloud data. The aforementioned fused ground point cloud data is obtained by fusing ground point cloud data. The aforementioned ground point cloud data represents the point cloud data of the ground. For example, a horizontal plane can be identified from the point cloud data and marked as the ground to obtain the ground point cloud data. Alternatively, Principal Component Analysis (PCA) can be used to segment the ground point cloud data; this disclosure does not specifically limit this method.

[0053] As one implementation method, when dividing the ground corresponding to the fused point cloud data according to a preset resolution, the ground corresponding to the fused ground point cloud data within a preset range can be divided according to the aforementioned preset resolution. For example, the ground corresponding to the fused point cloud data within a range of 16m*16m can be divided according to the aforementioned resolution of 0.03125m, resulting in 512*512 pixel grids, where each pixel grid represents a ground patch of size 0.03125m*0.03125m.

[0054] Then, each ground patch can be used as a target ground patch, and the above steps S101-S106 can be performed to apply texture mapping to each ground patch, thereby obtaining the texture map of each ground patch.

[0055] Alternatively, the entire ground coordinates can be pre-divided according to the aforementioned preset resolution to obtain multiple ground patches. The entire ground is larger than the ground corresponding to the fused ground point cloud data. The ground patches corresponding to each point cloud data marked as ground in the fused point cloud data are then used as target ground patches.

[0056] Since the aforementioned target ground patch is obtained by dividing the ground corresponding to the fused point cloud data based on a preset resolution, that is, by dividing the ground corresponding to the fused point cloud data according to a regular grid, the ground coordinates of the aforementioned target ground patch can be calculated based on the ground coordinates and the aforementioned preset resolution. The aforementioned ground coordinates are two-dimensional planar coordinates.

[0057] In high-precision map construction scenarios, the coordinates of each point cloud data point in the fused point cloud data correspond one-to-one with the road coordinates in the actual space. Therefore, as a specific implementation method, the ground coordinates corresponding to any point cloud data point within the aforementioned preset range can be obtained, and the coordinates of each ground patch within that preset range can be calculated based on the ground coordinates and a preset resolution. For example, for the ground corresponding to point cloud data within a 16m*16m range, the ground coordinates corresponding to the point cloud data at the four vertices can be obtained. Then, the coordinates of the target ground patch can be calculated based on the ground coordinates of the four vertices, according to the location of the target ground patch. Figure 2 As shown, in a predefined coordinate system, if the coordinates of the four ground vertices are (0m, 0m), (0m, 16m), (16m, 0m), and (16m, 16m), and the pixel position of the ground texture map within the preset range of the target ground patch is (100pt, 100pt), with each pixel representing a 0.03125m * 0.03125m area of ​​ground, then the coordinates of the target ground patch are (3.125m, 3.125m). The predefined coordinate system mentioned above is the coordinate system used by the fused point cloud data. It can be an ENU (East is the X-axis, North is the Y-axis, and Altitude is the Z-axis) coordinate system, which can be set according to actual needs.

[0058] As described above, texture mapping projects points from a 3D object or scene onto a 2D image to obtain information about the 3D object or scene from the image. Therefore, the spatial coordinates of the target ground patch obtained in step S101 should be 3D coordinates. The 2D planar coordinates of the target ground patch can be obtained through the above process. Therefore, based on the 2D planar coordinates of the target ground patch, the elevation of the road location matching these 2D coordinates can be obtained from the elevation information of the fused point cloud data, and together with the 2D planar coordinates of the target ground patch, constitutes the spatial coordinates of the target ground patch.

[0059] As one specific implementation method, such as Figure 3a As shown, the elevation information of the fused point cloud data can be obtained through the following steps:

[0060] Step S301: For each point cloud data acquisition point, obtain the plane formed by the point cloud data within a preset range of the point cloud data acquisition point;

[0061] Step S302: Based on the elevation of each point cloud data acquisition point and each plane, an initial point cloud elevation grid is obtained;

[0062] Step S303: Optimize the initial point cloud elevation grid according to the plane constraint function and the smoothing constraint function to obtain the target point cloud elevation grid. The plane constraint function is used to align the planes whose elevation differences between the corresponding point cloud data acquisition points are less than a preset threshold, and the smoothing constraint is used to make the connection between the planes smooth.

[0063] The following is an exemplary description of steps S301-S303 above:

[0064] In practical applications, the acquired point cloud data typically includes three-dimensional coordinates: x-coordinate (X), y-coordinate (Y), and height coordinate (Z), as well as acquisition time and acquisition pose. Acquisition pose refers to the pose of the point cloud data acquisition device, such as radar or a binocular camera, during point cloud data acquisition, including the device's position and orientation. The position of the acquisition device is the acquisition point of the point cloud data. Therefore, in step S301, the acquisition point information of each point cloud data point can be obtained from each point cloud data point, and a plane can be constructed within a second preset range of that acquisition point to obtain multiple planes. The second preset range can be set according to actual needs. For example, it can be a range of 1m or 2m from the point cloud data acquisition point.

[0065] After obtaining the aforementioned multiple planes, these planes can be merged and layered according to the elevation of the corresponding acquisition points during the construction of each plane. For example, planes with elevation differences less than a preset threshold can be merged into the same layer, while planes with elevation differences greater than the preset threshold can be divided into different layers. The point cloud data is then meshed to obtain an initial elevation mesh. This initial elevation mesh can be a triangular mesh or a quadrilateral mesh; this disclosure does not specifically limit its use. The preset threshold can be set according to actual needs.

[0066] The initial point cloud elevation grid can then be optimized based on preset planar constraint functions and smoothing constraint functions to obtain a more accurate elevation grid, thereby improving the accuracy of subsequent elevation acquisition. The aforementioned planar constraint functions are used to align the elevation differences between the corresponding point cloud data acquisition points to each plane that are less than the aforementioned preset threshold, and the aforementioned smoothing constraints are used to ensure smooth connections between the planes.

[0067] The following example, using a triangular mesh, illustrates the optimization process described above. A triangular mesh is a mesh in which each cell is divided into four triangular facets by a grid point and a center point, such as... Figure 3b As shown, each triangular facet is a mesh optimization unit.

[0068] The aforementioned plane constraint function and smoothing constraint function can be set according to actual needs. For example, the plane constraint function `cost_plane` and the smoothing constraint function `cost_smooth` can be:

[0069]

[0070]

[0071] In the planar constraint function, k is the number of point clouds within the triangular patch, n is the normal vector of the triangular patch, calculated from the coordinates of the three vertices v1, v2, and v3 of the current height grid triangular patch, and x... iLet z be the coordinates of the i-th point cloud data within the aforementioned triangular patch. In the smoothing constraint function, m is the number of elevations to be optimized, and z... i Let z be the elevation of the grid points or center point of the current grid to be optimized. mean The elevation is the average of four points within the third preset range of the current point cloud data to be optimized. The aforementioned third preset range can be set according to actual needs, and this disclosure does not impose specific limitations on it.

[0072] After optimizing the initial point cloud height grid according to the aforementioned planar constraint function and smoothing constraint function, i.e., minimizing the aforementioned planar constraint function and smoothing constraint function, the target point cloud height grid can be obtained. The height grid of the target point cloud can store the elevation of each grid point and the center point. The elevation of the corresponding position can be accessed in the height grid using any planar coordinate (x, y).

[0073] Therefore, the elevation data of the target ground patch can be obtained from the target point cloud height grid based on the two-dimensional coordinates of the target ground patch. The two-dimensional coordinates and elevation data of the target ground patch constitute the spatial coordinates of the target ground patch.

[0074] In high-precision map scenarios, due to the presence of multi-layered structures such as overpasses, multiple elevation data points may be obtained for the same two-dimensional coordinates. In other words, the two-dimensional coordinates of the aforementioned target ground patch may mark multiple road locations. Therefore, the two-dimensional coordinates of the target ground patch and its corresponding elevations can be used separately as the spatial coordinates of the target ground patch for subsequent calculations.

[0075] Of course, for each elevation in the fused point cloud data, the road surface corresponding to that elevation can be divided according to a preset resolution to obtain the ground surface patch of each road surface layer, and the three-dimensional coordinates of each ground surface patch can be obtained.

[0076] In practical applications, point cloud data acquisition and image acquisition devices are typically carried out simultaneously using the same drone, data acquisition vehicle, or other equipment. The point cloud data acquisition device can be a LiDAR (Light Detection and Ranging) system, a binocular camera, etc. The image acquisition device is usually a rolling shutter camera. The pose transformation relationship between the point cloud data acquisition device and the image acquisition device is usually determined in advance. Specifically, the pose of the image acquisition device at the same moment can be obtained by transforming the pose of the point cloud data acquisition device based on the camera extrinsic parameters. These camera extrinsic parameters are usually determined before use.

[0077] The following example uses a LiDAR as the point cloud data acquisition device and an airborne rolling shutter camera as the image acquisition device to illustrate the texture mapping method provided in this disclosure.

[0078] After determining the spatial coordinates of the target ground patch to be mapped, an image containing those coordinates can be determined. There can be one or more images containing the target ground patch; this disclosure does not impose any specific limitations on this.

[0079] Then, the corresponding image poses of the above images can be obtained, that is, the poses of the rolling shutter camera when acquiring the above images. Specifically, based on the acquisition time of the image, the pose of the point cloud data acquisition device corresponding to the acquisition time can be obtained; and the pose transformation of the point cloud data acquisition device can be performed according to the camera extrinsic parameters for acquiring the image to obtain the image pose of the image.

[0080] For example, if the image acquisition time is t1, the pose of the radar that acquired the point cloud data at time t1 can be obtained, and the pose can be transformed according to the camera extrinsic matrix to obtain the image pose of the above image.

[0081] As a specific implementation method, the radar pose can also be corrected, and the corrected radar pose corresponding to the above acquisition time can be obtained to further improve the projection accuracy.

[0082] Because rolling shutter cameras acquire images through line-by-line exposure, when a drone or data acquisition vehicle carrying a rolling shutter camera moves and acquires image data, the camera pose in each line of the resulting image will be different. Figure 4 As shown, Figure 4 The exposure mechanism of a rolling shutter is shown:

[0083] The actual time it takes for a rolling shutter camera to capture one frame of an image includes the exposure time and the image output time. Specifically, a rolling shutter camera captures an image frame using a line-by-line exposure method, meaning that the exposure time and image output time are different for each row of pixels in the image.

[0084] However, the time recorded by a rolling shutter camera for capturing one frame refers to the time between the end of exposure of the first row of pixels in the previous frame and the end of exposure of the first row of pixels in the current frame. The difference between the image output time of the last row of pixels in the previous frame and the end of exposure of the first row of pixels in the current frame is the acquisition delay time between the two frames. In other words, the image acquisition time recorded by a rolling shutter camera is actually the acquisition time of the first row of pixels in the image.

[0085] Therefore, after performing extrinsic parameter transformation on the radar pose using camera extrinsic parameters, the resulting image pose is the image pose corresponding to the first row of pixels in the image. Therefore, in this disclosure, the image pose is referred to as the first row pose. Since the pose of the point cloud data acquisition device and the image acquisition time can be directly obtained from the point cloud data and the image, the accurate first row pose can be obtained through the above steps. However, the projection point corresponding to the target ground patch may not fall in the first row of the image. Therefore, the row pose of the projection point of the target ground patch in the image can be calculated based on the first row pose.

[0086] As one specific implementation, the pose of the target ground patch's projection row in the image can be estimated iteratively for each image. Specifically, such as... Figure 5 As shown, the following steps may be included:

[0087] Step S501: Based on the spatial coordinates of the target ground patch and the pose of the first row of the image, determine the initial projection point of the target ground patch in the image;

[0088] Step S502: Determine the image pose corresponding to the initial projection point as a candidate image pose;

[0089] Step S503: Based on the spatial coordinates of the target ground patch and the pose of the candidate image, determine the current projection point of the target ground patch in the image;

[0090] Step S504: Determine whether the preset convergence condition has been met. If the preset convergence condition has not been met, proceed to step S505; if the preset convergence condition has been met, proceed to step S506.

[0091] Step S505: Determine the image pose corresponding to the current projection point as a new candidate image pose, and return to step S503;

[0092] Step S506: Determine the image pose corresponding to the current projection point as the target pose.

[0093] The following is an exemplary description of steps S501-S506 above:

[0094] In one specific implementation, in step S501, the coordinates of the initial projection point can be obtained according to the following formula:

[0095] x=π(K[R1 t1]X) Formula 1

[0096] In Formula 1, X and x represent the spatial coordinates of the target ground patch and the coordinates of the corresponding projected point, respectively. K is the camera intrinsic parameter matrix, π is a parameter reflecting lens distortion, and [R1 t1] is the pose of the first row of the image, which can be represented using quaternions. The projected point coordinates are uv coordinates. Any pixel on the image can be located using two-dimensional uv coordinates. Here, u is the column coordinate in the horizontal direction, and v is the row coordinate in the vertical direction.

[0097] Then, the image pose corresponding to the coordinates of the initial projection points can be calculated.

[0098] One approach is to extract feature point coordinates from the image, construct a transformation matrix based on the image feature point coordinates and the corresponding target ground patch spatial coordinates, and then obtain the image pose corresponding to the aforementioned projection point coordinates based on this transformation matrix. However, this method is affected by the feature point selection method, resulting in lower accuracy and less flexibility.

[0099] In another specific implementation, this disclosure assumes that the radar and rolling shutter camera move at a constant speed during the acquisition of point cloud data and images. Therefore, if the image pose at the start of acquisition and the image pose at the end of acquisition are known, the image pose corresponding to any pixel in the image can be estimated by interpolating the pose of the first row and the pose of the last row of the image. That is, the pose of the rolling shutter camera in the row of any pixel in the acquired image can be estimated, improving the accuracy and flexibility of obtaining the coordinates of the projection point.

[0100] For example, the candidate image pose corresponding to the above projection point coordinates can be obtained through the following steps:

[0101] Step S1: Obtain the row coordinates of the current projection point.

[0102] Step S2: Perform interpolation calculation on the first row pose and the last row pose of the image to obtain the image pose corresponding to the current projection point, which is used as a new candidate image pose.

[0103] The tail pose of the image can be calculated based on the first-row pose of the image. As a specific implementation, step S2 may include:

[0104] Step S21: Based on the image acquisition time and camera exposure time, obtain the image tail acquisition time;

[0105] Step S22: Based on the image acquisition time and the camera exposure time, obtain the image tail acquisition time;

[0106] Step S23: Obtain the image tail position based on the image tail acquisition time.

[0107] The camera exposure time mentioned above can be obtained from the camera parameters. The image tail acquisition time mentioned above is the sum of the image acquisition time and the camera exposure time, t + Δt.

[0108] Then, the radar pose at that time can be obtained based on the image tail acquisition time, and the radar pose can be transformed using camera extrinsic parameters to obtain the image tail pose. In this way, by calculating the image tail pose based on the accurately obtainable first-line pose of the image and the camera exposure time, the tail pose can be accurately obtained, thus yielding accurate interpolation results.

[0109] The movement of radar and rolling shutter cameras can generally be decomposed into rotation and translation. Therefore, interpolating the pose of the first and last rows of an image can include interpolation in the rotational direction and interpolation in the translational direction, thereby obtaining a more accurate row pose of the projection point.

[0110] As one specific implementation method, the first row pose and the last row pose of the image can be interpolated using the following formula.

[0111]

[0112] In this formula, (q) last_ ,t last_ ) represents the pose of the first row in the coordinate system of the last row, which can be obtained based on the pose of the first row and the pose of the last row of the image. (q) last_ ,t last_ Let be the pose of the v-th row in the tail coordinate system, and r be the row coefficient corresponding to the projection point coordinates, specifically r = v / h, where h is the image height. Alternatively, the row coefficient can be the product of the ratio of the projection point row coordinates v to the image height h and a preset coefficient. This disclosure does not specifically limit this. q1 and t1 are preset initial poses, q1 = (1,0,0,0) and t1 = (0,0,0). The pose of the v-th row in the world coordinate system can then be obtained through coordinate system transformation.

[0113] After obtaining the above candidate image poses, the target ground patch can be reprojected according to Formula 1 based on the above candidate image poses and the spatial coordinates of the target ground patch to obtain the new current projection point coordinates. Based on the row coordinate v in the projection point coordinates, the row pose of the current projection point can be recalculated according to Formula 2.

[0114] Repeat the above process until a preset convergence condition is met. This preset convergence condition can be set according to actual needs. For example, it can be that the difference between the row coordinates v of the two obtained projection points is less than a preset threshold. This preset threshold can be set according to actual needs, for example, it can be 0.5 pixels. The convergence condition can also be that the poses of the two obtained images are less than a preset pose threshold, etc. This disclosure does not impose specific limitations on this.

[0115] When the preset convergence condition is met, the image pose corresponding to the current projection point can be used as the target pose. By using interpolation to obtain the image pose of the projection point coordinates of the target ground patch in the image based on the first and last row poses of the image, the accuracy of the obtained image pose corresponding to the target ground patch is improved, thereby improving the subsequent projection accuracy.

[0116] After obtaining the target pose of the row where the projection point of the target ground patch is located, the target projection point coordinates of the target ground patch in the above image can be calculated for each image using Formula 1 above, based on the spatial coordinates of the target ground patch and the target pose.

[0117] After obtaining the target projection point coordinates, the texture information of the corresponding pose pixels can be copied to the target ground patch according to the target projection point coordinates.

[0118] Since images captured by a rolling shutter camera may contain occlusions of ground features such as vehicles and pedestrians, as a specific implementation method, the target ground surface can be mapped using the following steps:

[0119] Step S601: Determine the target image from each of the images where the target projection point is a ground feature.

[0120] For example, target detection can be performed on the images to determine whether the target projection points of the target ground patch in each image are occluded by vehicles or pedestrians. Then, any image without occlusion can be selected as the target image. Alternatively, an image with a resolution higher than a preset resolution threshold can be selected as the target image to further improve texture mapping accuracy.

[0121] Step S602: Project the texture information at the target projection point in the target image onto the target ground patch.

[0122] In step S602, the RGB values ​​of the projection points of the target ground patch in the target image can be assigned to the pixels where the target ground patch is located in the texture map. After texture mapping is performed on each ground patch, the texture map of the ground corresponding to the above-mentioned fused ground point cloud data can be obtained.

[0123] like Figure 6 As shown, Figure 6 This is an execution framework diagram of the texture mapping method provided in this disclosure. It mainly includes three parts: elevation mesh construction, rolling shutter model interpolation, and ground texture projection.

[0124] The elevation grid construction specifically includes: 1. Constructing a trajectory plane based on the corrected radar pose and fused ground point cloud data. That is, constructing a plane according to the elevation of each ground point cloud data acquisition point. 2. Merging and layering the plane to obtain an initial point cloud elevation grid. 3. Optimizing the initial point cloud elevation grid to obtain the target elevation grid.

[0125] The rolling shutter model interpolation specifically includes: obtaining the first-row pose and the last-row pose of the image based on the corrected radar pose, each image including the target ground patch, the timestamp of each image, and camera parameters. The aforementioned camera parameters include exposure time, camera extrinsic matrix, and camera intrinsic matrix. Interpolation is then performed on the first-row pose and the last-row pose to obtain the image pose of the row containing the target ground patch projection points in each image.

[0126] The ground texture projection part includes: 1. Ground patch division, i.e., dividing the ground into patches according to a preset resolution. 2. Obtaining the elevation of the target ground patch from the target elevation grid, thus obtaining the spatial coordinates of the target ground patch. 3. Projecting the target ground patch onto the image, i.e., projecting the target ground patch onto the image based on the spatial coordinates of the target ground patch and the image pose of the row where the projection point of the target ground patch is located, thus obtaining the projection point of the target ground patch in each image. The above candidate images are images containing the target ground patch. 4. Image filtering. That is, selecting the best image from all images, such as images where the ground is not obscured by vehicles, pedestrians, etc., and has high clarity. Assigning the RGB value of the pixel corresponding to the projection point coordinates in the best image to the pixel position of the target ground patch in the texture map.

[0127] By applying the embodiments of this disclosure, a rolling shutter imaging model is constructed based on the ground texture mapping framework of the rolling shutter camera model, and the ground texture mapping is performed using this model, which effectively corrects the texture mapping drift caused by rolling shutter exposure.

[0128] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0129] According to embodiments of this disclosure, this disclosure also provides an apparatus for texture mapping, such as... Figure 7 As shown, the above-mentioned device may include:

[0130] The spatial coordinate acquisition module 701 is used to acquire the spatial coordinates of the target ground patch based on the fused point cloud data, wherein the ground patch is obtained by dividing the ground corresponding to the fused point cloud data based on a preset resolution;

[0131] Image acquisition module 702 is used to acquire an image including the target ground patch based on the spatial coordinates of the target ground patch;

[0132] The first-row pose acquisition module 703 is used to acquire the first-row pose of the image, wherein the first-row pose is the pose of the camera when acquiring the image;

[0133] The target pose estimation module 704 is used to estimate the target pose of the row in which the projection point of the target ground patch is located in the image based on the pose of the first row of the image.

[0134] The target projection point determination module 705 is used to determine the target projection point of the target ground patch in the image based on the spatial coordinates of the target ground patch and the target pose;

[0135] The texture mapping module 706 is used to perform texture mapping on the target ground patch based on the target projection point of the target ground patch in the image.

[0136] In one possible embodiment, the target pose estimation module is used to determine the initial projection point of the target ground patch in the image based on the spatial coordinates of the target ground patch and the pose of the first row of the image;

[0137] The image pose corresponding to the initial projection point is determined as a candidate image pose;

[0138] Based on the spatial coordinates of the target ground patch and the pose of the candidate image, determine the current projection point of the target ground patch in the image;

[0139] Determine the image pose corresponding to the current projection point as a new candidate image pose, and return the step of determining the current projection point of the target ground patch in the image based on the spatial coordinates of the target ground patch and the candidate image pose, until the preset convergence condition is reached;

[0140] The image pose corresponding to the current projection point is determined as the target pose.

[0141] In one possible embodiment, determining the image pose corresponding to the current projection point as a new candidate image pose includes:

[0142] Obtain the row coordinates of the current projection point;

[0143] Interpolation calculations are performed on the first row pose and the last row pose of the image to obtain the image pose corresponding to the current projection point, which is then used as a new candidate image pose.

[0144] In one possible embodiment, the step of interpolating the first row pose and the last row pose of the image to obtain the image pose corresponding to the current projection point includes:

[0145] The first row pose and the last row pose of the image are subjected to rotational interpolation and linear interpolation to obtain the image pose corresponding to the current projection point.

[0146] In one possible embodiment, the target pose estimation module is used to obtain the image tail-hunting time based on the image acquisition time and the camera exposure time.

[0147] Based on the image tail acquisition time, the image tail pose is obtained.

[0148] In one possible embodiment, obtaining the first row pose of the image includes:

[0149] Based on the acquisition time of the image, obtain the pose of the point cloud data acquisition device corresponding to the acquisition time;

[0150] The pose of the point cloud data acquisition device is transformed according to the camera extrinsic parameters used to acquire the image, thereby obtaining the first row pose of the image.

[0151] In one possible embodiment, the above-described apparatus may further include:

[0152] The point cloud data fusion module is used to obtain a plane composed of point cloud data within a preset range of each point cloud data acquisition point.

[0153] Based on the elevation of each point cloud data acquisition point and each plane, an initial point cloud elevation grid is obtained;

[0154] The initial point cloud elevation grid is optimized according to the plane constraint function and the smoothing constraint function to obtain the target point cloud elevation grid. The plane constraint function is used to align the planes whose elevation differences between the corresponding point cloud data acquisition points are less than a preset threshold, and the smoothing constraint is used to make the connection between the planes smooth.

[0155] The process of obtaining the spatial coordinates of the target ground patch based on the fused point cloud data includes:

[0156] Obtain the planar coordinates of the target ground patch;

[0157] Based on the planar coordinates of the target ground patch, the elevation data of the target ground patch is obtained from the elevation grid of the target point cloud, and the spatial coordinates of the target ground patch are obtained.

[0158] In one possible embodiment, there are multiple images including the target ground patch;

[0159] The process of performing texture mapping on the target ground patch based on the target projection points of the target ground patch in the image includes:

[0160] The target image at the target projection point is determined from each of the images to be a ground feature;

[0161] The texture information at the target projection point in the target image is projected onto the target ground patch.

[0162] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0163] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0164] like Figure 8 As shown, device 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 802 or a computer program loaded from storage unit 808 into random access memory (RAM) 803. RAM 803 may also store various programs and data required for the operation of device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via bus 804. Input / output (I / O) interface 805 is also connected to bus 804.

[0165] Multiple components in device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of monitors, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0166] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as the texture mapping method. For example, in some embodiments, the texture mapping method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the texture mapping method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the texture mapping method by any other suitable means (e.g., by means of firmware).

[0167] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0168] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0169] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0170] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0171] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0172] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0173] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0174] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for texture mapping, comprising: The spatial coordinates of the target ground patch are obtained based on the fused point cloud data, wherein the ground patch is obtained by dividing the ground corresponding to the fused point cloud data based on a preset resolution; Based on the spatial coordinates of the target ground patch, obtain an image including the target ground patch; Obtain the first row pose of the image, wherein the first row pose is the pose of the camera when it captures the image; Based on the pose of the first row of the image, the target pose of the target ground patch in the image is estimated to be the row in which the projection point of the target ground patch is located. Based on the spatial coordinates of the target ground patch and the target pose, determine the target projection point of the target ground patch in the image; Texture mapping is performed on the target ground patch based on the target projection points of the target ground patch in the image; Wherein, the step of estimating the target pose of the row containing the projection point of the target ground patch in the image based on the pose of the first row of the image includes: Based on the spatial coordinates of the target ground patch and the pose of the first row of the image, the initial projection point of the target ground patch in the image is determined; The image pose corresponding to the initial projection point is determined as a candidate image pose; Based on the spatial coordinates of the target ground patch and the pose of the candidate image, determine the current projection point of the target ground patch in the image; Determine the image pose corresponding to the current projection point as a new candidate image pose, and return the step of determining the current projection point of the target ground patch in the image based on the spatial coordinates of the target ground patch and the candidate image pose, until the preset convergence condition is reached; The image pose corresponding to the current projection point is determined as the target pose.

2. The method according to claim 1, wherein, Determining the image pose corresponding to the current projection point as a new candidate image pose includes: Obtain the row coordinates of the current projection point; Interpolation calculations are performed on the first row pose and the last row pose of the image to obtain the image pose corresponding to the current projection point, which is then used as a new candidate image pose.

3. The method according to claim 2, wherein, The step of interpolating the first row pose and the last row pose of the image to obtain the image pose corresponding to the current projection point includes: The first row pose and the last row pose of the image are subjected to rotational interpolation and linear interpolation to obtain the image pose corresponding to the current projection point.

4. The method according to claim 2, further comprising: Based on the image acquisition time and the camera exposure time, the image tail acquisition time is obtained; Based on the image tail acquisition time, the image tail pose is obtained.

5. The method according to claim 1, wherein, The step of obtaining the first row pose of the image includes: Based on the acquisition time of the image, obtain the pose of the point cloud data acquisition device corresponding to the acquisition time; The pose of the point cloud data acquisition device is transformed according to the camera extrinsic parameters used to acquire the image, thereby obtaining the first row pose of the image.

6. The method according to claim 1, further comprising: For each point cloud data acquisition point, obtain the plane formed by the point cloud data within a preset range of the point cloud data acquisition point; Based on the elevation of each point cloud data acquisition point and each plane, an initial point cloud elevation grid is obtained; The initial point cloud elevation grid is optimized according to the plane constraint function and the smoothing constraint function to obtain the target point cloud elevation grid. The plane constraint function is used to align the planes whose elevation differences between the corresponding point cloud data acquisition points are less than a preset threshold, and the smoothing constraint is used to make the connection between the planes smooth. The process of obtaining the spatial coordinates of the target ground patch based on the fused point cloud data includes: Obtain the planar coordinates of the target ground patch; Based on the planar coordinates of the target ground patch, the elevation data of the target ground patch is obtained from the elevation grid of the target point cloud, and the spatial coordinates of the target ground patch are obtained.

7. The method according to claim 1, wherein, There are multiple images including the target ground patch; The process of performing texture mapping on the target ground patch based on the target projection points of the target ground patch in the image includes: The target image at the target projection point is determined from each of the images to be a ground feature; The texture information at the target projection point in the target image is projected onto the target ground patch.

8. A texture mapping apparatus, comprising: The spatial coordinate acquisition module is used to acquire the spatial coordinates of the target ground patch based on the fused point cloud data, wherein the ground patch is obtained by dividing the ground corresponding to the fused point cloud data based on a preset resolution; The image acquisition module is used to acquire an image including the target ground patch based on the spatial coordinates of the target ground patch; The first-row pose acquisition module is used to acquire the first-row pose of the image, wherein the first-row pose is the pose of the camera when acquiring the image; The target pose estimation module is used to estimate the target pose of the row in which the projection point of the target ground patch is located in the image based on the pose of the first row of the image. The target projection point determination module is used to determine the target projection point of the target ground patch in the image based on the spatial coordinates of the target ground patch and the target pose; The texture mapping module is used to perform texture mapping on the target ground patch based on the target projection points of the target ground patch in the image; The target pose estimation module is configured to: determine the initial projection point of the target ground patch in the image based on the spatial coordinates of the target ground patch and the first row pose of the image; determine the image pose corresponding to the initial projection point as a candidate image pose; determine the current projection point of the target ground patch in the image based on the spatial coordinates of the target ground patch and the candidate image pose; determine the image pose corresponding to the current projection point as a new candidate image pose; return to the step of determining the current projection point of the target ground patch in the image based on the spatial coordinates of the target ground patch and the candidate image pose, until a preset convergence condition is met; and determine the image pose corresponding to the current projection point as the target pose.

9. The apparatus according to claim 8, wherein, Determining the image pose corresponding to the current projection point as a new candidate image pose includes: Obtain the row coordinates of the current projection point; Interpolation calculations are performed on the first row pose and the last row pose of the image to obtain the image pose corresponding to the current projection point, which is then used as a new candidate image pose.

10. The apparatus according to claim 9, wherein, The step of interpolating the first row pose and the last row pose of the image to obtain the image pose corresponding to the current projection point includes: The first row pose and the last row pose of the image are subjected to rotational interpolation and linear interpolation to obtain the image pose corresponding to the current projection point.

11. The apparatus according to claim 9, wherein, The target pose estimation module is used to obtain the image tail acquisition time based on the image acquisition time and the camera exposure time. Based on the image tail acquisition time, the image tail pose is obtained.

12. The apparatus according to claim 8, wherein, The step of obtaining the first row pose of the image includes: Based on the acquisition time of the image, obtain the pose of the point cloud data acquisition device corresponding to the acquisition time; The pose of the point cloud data acquisition device is transformed according to the camera extrinsic parameters used to acquire the image, thereby obtaining the first row pose of the image.

13. The apparatus according to claim 8, further comprising: The point cloud data fusion module is used to obtain a plane composed of point cloud data within a preset range of each point cloud data acquisition point. Based on the elevation of each point cloud data acquisition point and each plane, an initial point cloud elevation grid is obtained; The initial point cloud elevation grid is optimized according to the plane constraint function and the smoothing constraint function to obtain the target point cloud elevation grid. The plane constraint function is used to align the planes whose elevation differences between the corresponding point cloud data acquisition points are less than a preset threshold, and the smoothing constraint is used to make the connection between the planes smooth. The process of obtaining the spatial coordinates of the target ground patch based on the fused point cloud data includes: Obtain the planar coordinates of the target ground patch; Based on the planar coordinates of the target ground patch, the elevation data of the target ground patch is obtained from the elevation grid of the target point cloud, and the spatial coordinates of the target ground patch are obtained.

14. The apparatus according to claim 8, wherein, There are multiple images including the target ground patch; The process of performing texture mapping on the target ground patch based on the target projection points of the target ground patch in the image includes: The target image at the target projection point is determined from each of the images to be a ground feature; The texture information at the target projection point in the target image is projected onto the target ground patch.

15. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-7.

17. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-7.

Images