Map generation method and apparatus, electronic device, and computer storage medium

Abstract

Embodiments of the present application provide a map generation method and device, electronic equipment and computer storage medium. The method comprises: obtaining a sequence of road images and a sequence of track points of a collection device of the road images, the track points being used to indicate the pose of the collection device in a world coordinate system when collecting the corresponding road image; obtaining a road surface area from the road image; obtaining a first spatial coordinate of a first pixel of the road surface area in a collection device coordinate system; determining a second spatial coordinate of a mapping point to which the first pixel is mapped in the world coordinate system according to the pose of the track point corresponding to the road surface area to which the first pixel belongs, so as to generate a point cloud map according to the second spatial coordinate. The method has better use effect.

Description

Technical Field

[0001] This application relates to the field of map technology, and in particular to a map generation method, apparatus, electronic device, and computer storage medium. Background Technology

[0002] In road maintenance and repair, regular inspections of road surface cracks are necessary. Current methods for inspecting road surface cracks typically involve using cameras to capture a series of images of the road surface and its surrounding environment, followed by manual inspection of these images for cracks. This approach suffers from drawbacks such as low detection efficiency and difficulty in accurately identifying the geographical location of cracks. Summary of the Invention

[0003] In view of this, embodiments of this application provide a map generation scheme to at least partially solve the above-mentioned problems.

[0004] According to a first aspect of the present application, a map generation method is provided, comprising: acquiring a sequence of road images and a sequence of trajectory points of a road image acquisition device, wherein the trajectory points are used to indicate the pose of the acquisition device in a world coordinate system when acquiring the corresponding road image; acquiring a road surface region from the road images; acquiring a first spatial coordinate of a first pixel of the road surface region in the coordinate system of the acquisition device; determining a second spatial coordinate of a mapping point in the world coordinate system to which the first pixel is mapped, based on the pose of the trajectory point corresponding to the road surface region to which the first pixel belongs, so as to generate a point cloud map based on the second spatial coordinate.

[0005] According to a second aspect of the embodiments of this application, a map generation apparatus is provided, comprising: a first acquisition module, configured to acquire a sequence of road images and a sequence of trajectory points of an acquisition device for the road images, the trajectory points indicating the pose of the acquisition device in a world coordinate system when acquiring the corresponding road images; a second acquisition module, configured to acquire a road surface region from the road images; a third acquisition module, configured to acquire a first spatial coordinate of a first pixel of the road surface region in the coordinate system of the acquisition device; and a determination module, configured to determine a second spatial coordinate of a mapping point in the world coordinate system corresponding to the first pixel based on the pose of the trajectory point corresponding to the road surface region to which the first pixel belongs, so as to generate a point cloud map based on the second spatial coordinate.

[0006] According to a third aspect of the present application, an electronic device is provided, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus; the memory is used to store at least one executable instruction, wherein the executable instruction causes the processor to perform an operation corresponding to the method described in the first aspect.

[0007] According to a fourth aspect of the embodiments of this application, a computer storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0008] The method provided in this application can construct a road surface map based on road images. The road surface map consists of a point cloud map and a two-dimensional map (i.e., a two-dimensional photo map) of the road surface. This method can perform coarse and fine calculations of the spatial coordinates of the first pixel of the road surface, providing a method for depth estimation of a monocular acquisition device. Furthermore, it can accurately obtain the geographical location of each second pixel based on the two-dimensional map of the road surface, enabling target location and measurement from the two-dimensional map. The two-dimensional map includes rich colors, thus compensating for the poor reflectivity of the point cloud in LiDAR. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0010] Figure 1 A schematic diagram of an exemplary system to which the methods of the embodiments of this application are applied;

[0011] Figure 2 This is a flowchart of the steps of a method according to Embodiment 1 of this application;

[0012] Figure 3 This is a flowchart of a sub-step of step S206 according to an embodiment of this application;

[0013] Figure 4A This is a schematic diagram of the imaging ray and global plane according to Embodiment 1 of this application;

[0014] Figure 4B This is a schematic diagram of the imaging rays, global plane, and local plane according to Embodiment 1 of this application;

[0015] Figure 5 This is a flowchart of another step of a method according to Embodiment 1 of this application;

[0016] Figure 6 This is a flowchart of the sub-steps of step S212 according to Embodiment 1 of this application;

[0017] Figure 7 This is a schematic diagram illustrating the conversion of an image sequence into a two-dimensional map according to Embodiment 1 of this application;

[0018] Figure 8 This is a structural block diagram of an apparatus according to Embodiment 2 of this application;

[0019] Figure 9 This is a schematic diagram of the structure of an electronic device according to Embodiment 3 of this application. Detailed Implementation

[0020] To enable those skilled in the art to better understand the technical solutions in the embodiments of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art should fall within the protection scope of the embodiments of this application.

[0021] The specific implementation of the embodiments of this application will be further described below with reference to the accompanying drawings.

[0022] Figure 1 An exemplary system for which an embodiment of the method of this application is applicable is shown. For example... Figure 1 As shown, the system 100 may include a server 102, a communication network 104, and / or one or more user devices 106. Figure 1 The example in the text shows multiple user devices.

[0023] Server 102 can be any suitable server for storing information, data, programs, and / or any other suitable type of content. In some embodiments, server 102 can perform any suitable function. For example, in some embodiments, server 102 can generate a three-dimensional point cloud map based on road images.

[0024] In some embodiments, communication network 104 may be any suitable combination of one or more wired and / or wireless networks. For example, communication network 104 may include any one or more of the following: the Internet, intranet, wide area network (WAN), local area network (LAN), wireless network, digital subscriber line (DSL) network, frame relay network, asynchronous transfer mode (ATM) network, virtual private network (VPN), and / or any other suitable communication network. User equipment 106 may be connected to communication network 104 via one or more communication links (e.g., communication link 112), and communication network 104 may be linked to server 102 via one or more communication links (e.g., communication link 114). Communication links may be any communication link suitable for transmitting data between user equipment 106 and server 102, such as network links, dial-up links, wireless links, hardwired links, any other suitable communication links, or any suitable combination of such links.

[0025] In some embodiments, user equipment 106 may include any suitable type of device. For example, in some embodiments, user equipment 106 may include mobile devices, tablet computers, laptop computers, desktop computers, wearable computers, game consoles, media players, vehicle entertainment systems, and / or any other suitable type of user equipment.

[0026] Although server 102 is illustrated as a single device, in some embodiments, any suitable number of devices may be used to perform the functions performed by server 102. For example, in some embodiments, multiple devices may be used to implement the functions performed by server 102. Alternatively, cloud services may be used to implement the functions of server 102.

[0027] like Figure 2 As shown, based on the above system, this application provides a method, which will be described below through several embodiments.

[0028] Step S202: Obtain a sequence of road images and a sequence of trajectory points of the road image acquisition device, wherein the trajectory points are used to indicate the pose of the acquisition device in the world coordinate system when acquiring the corresponding road image.

[0029] The sequence of road images includes multiple road images. The acquisition device can be a camera or other device capable of acquiring images. The sequence of trajectory points of the acquisition device includes trajectory points at multiple different times, and each trajectory point indicates the pose of the acquisition device in the world coordinate system at that time.

[0030] For example, in a sequence of road images, there is a correspondence between road images and trajectory points, and the pose of the trajectory points indicates the pose of the acquisition device in the world coordinate system when the road image was captured.

[0031] Step S204: Obtain the road surface area from the road image.

[0032] To avoid obscuring the road surface from trees, vehicles, or signs in road images, thus preventing a clear and comprehensive view of the road surface, a section of the road surface is extracted from the road image.

[0033] For example, step S204 can be implemented as follows: identifying the road edge from the road image; and extracting the road surface area from the road image based on the identified road edge.

[0034] The identification of road edges can be achieved using either a trained neural network model or image processing algorithms, with no limitation on the method. Based on the identified road edges, the area between the road edges is extracted as the road surface area, thereby removing elements such as the median strip on both sides of the road.

[0035] Step S206: Obtain the first spatial coordinates of the first pixel of the road surface area in the coordinate system of the acquisition device.

[0036] Since the first pixel in the road surface area is a two-dimensional image, its three-dimensional coordinates in the acquisition device's coordinate system cannot be directly determined from the first pixel. To map the two-dimensional pixels to the three-dimensional coordinate system of the acquisition device, depth estimation is performed on the pixels in the road surface area. Depth estimation refers to estimating the distance from the acquisition device to various points in the scene, thereby enabling the reconstruction of a 3D point cloud from a 2D image.

[0037] In one feasible approach, such as Figure 3 As shown, step S206 includes the following sub-steps:

[0038] Sub-step S2061: Obtain the ground equation of the road surface area in the coordinate system of the acquisition device.

[0039] Typically, the ground (i.e., the road surface in this embodiment) can be equivalent to a plane. The relationship between the data acquisition device and the road surface can be represented by the normal vector of the road surface and the distance from the data acquisition device to the road surface. Its ground equation can be expressed as: Ax+y+Cz+D=0.

[0040] Where A, B, and C are the normal vectors of the road surface in the coordinate system of the data acquisition device, and D is the distance from the data acquisition device to the road surface.

[0041] It should be noted that since the road surface itself is not actually an ideal plane, but rather uneven, the ground equation includes at least a global equation, and may or may not include local equations as needed. The global equation represents the plane covering the entire road surface, while the local equation represents the plane covering only a portion of the road surface.

[0042] This embodiment provides several feasible methods for obtaining ground equations.

[0043] In the first approach, the ground equations can be obtained based on the sequence of acquired road images and the sequence of trajectory points. This approach allows for the acquisition of the global equations within the ground equations.

[0044] This method includes the following procedures:

[0045] Process A1: Identify feature points from the road image.

[0046] Feature points include, but are not limited to, the corner points of lane lines, zebra crossings, arrows, and the endpoints of the edge lines of poles along the road, but are not limited to these. Furthermore, it should be noted that the feature points in this embodiment are not limited to points; they can also be edge lines, etc.

[0047] Feature points can be identified using a trained neural network model or through image processing algorithms; there are no restrictions on this.

[0048] Process B1: Associate feature points in two adjacent road images with corresponding points to obtain pairs of corresponding points.

[0049] The corresponding point pair includes two corresponding points. A corresponding point refers to the same feature point on the same object in different road images. For example, road image 1 contains a corner point 'a' on arrow A, and road image 2 contains a corner point 'a' on arrow A'. Since arrow A and arrow A' are actually the same arrow, and corner point 'a' is the same corner point appearing in different road images, corner point 'a' in road image 1 and corner point 'a' in road image 2 can be associated as corresponding points to obtain a corresponding point pair.

[0050] Process C1: Based on the initialized ground equations and the pose of the trajectory points of the acquisition device, determine the residuals of the two corresponding points in the world coordinate system.

[0051] The parameters (A, B, C, and D) in the initialized ground equations can be randomly initialized. Based on the pixel positions of corresponding points in the corresponding road images and the initialized ground equations, the first spatial coordinates of the corresponding points in the coordinate system of the acquisition device can be calculated. The second spatial coordinates of the corresponding points in the world coordinate system can be determined based on the pose of the trajectory points corresponding to the road images to which the corresponding points belong. The residuals can then be calculated based on the second spatial coordinates of the two corresponding points in the world coordinate system.

[0052] Process D1: Based on the residual, at least the initialized ground equation is optimized to obtain the optimized ground equation as a global equation covering the road surface area.

[0053] Based on the calculated residuals, methods such as bundle adjustment and nonlinear optimization can be used to collaboratively optimize the pose of the trajectory points of the acquisition device and the initial ground orientation, thereby obtaining more accurate poses of the trajectory points and more accurate parameters of the global equation, thus obtaining an optimized global equation.

[0054] In the second approach, for cases with point clouds collected by lidar, ground equations can be obtained based on the collected point clouds. This approach yields both global and local equations within the ground equations.

[0055] Among them, the global equation can be optimized based on the pose of the acquired point cloud to obtain the global equation with the minimum sum of distances from all points in the acquired point cloud to the plane indicated by the global equation.

[0056] For local equations, this approach includes the following steps:

[0057] Process A2: Acquire the laser point cloud collected by the lidar.

[0058] Process B2: Obtain the ground point cloud from the laser point cloud.

[0059] To avoid the adverse effects of points other than ground points on the accuracy of local equations, the ground point cloud can be determined based on the height of each point's pose in the laser point cloud.

[0060] Process C2: Perform rasterization on the road surface covered by the ground point cloud and initialize the local equations corresponding to the raster.

[0061] The shape and size of the grid can be determined as needed, and this embodiment does not impose any restrictions on this. For example, the grid can be rectangular, and the length and width of the rectangle can be appropriately determined based on the length and width of the road surface, etc. A local equation can be initialized for each divided grid. The local equations initialized for different grids can be the same or different.

[0062] Process D2: Based on the pose of the ground points contained in each grid in the grid sequence, optimize the local equation of each grid to obtain the optimized local equation of the grid.

[0063] In one example, to improve fitting accuracy and ensure the continuity and smoothness of the planes indicated by adjacent local equations, process D2 can be implemented as follows: based on the pose of the ground points contained in each grid and the local equations of each grid, calculate the first distance from each ground point to the ground of its respective grid, the second distance between the grounds of two adjacent grids, and the angle between the grounds of two adjacent grids, wherein the ground is determined according to the local equations; optimize the local equations of each grid to obtain the local equation that minimizes the sum of the first distance, the sum of the second distance, and the sum of the angles corresponding to each grid as the local equation of the grid.

[0064] This optimization process can be represented as:

[0065]

[0066] Where {Plane set} is the set of parameters of the local equations to be optimized. Each element in the set represents the parameters of a local equation, which can be represented as (A, B, C, D).

[0067] Indicates the first distance, where Plane i Let p be the local equation corresponding to the i-th grid. jLet j be the j-th point in the i-th grid.

[0068] λ1 and λ2 are hyperparameters that can be set as needed.

[0069] This represents the second distance between the plane indicated by the local equation of the i-th grid and the plane indicated by the local equation of the adjacent preceding or following grid.

[0070] This represents the angle between the plane indicated by the local equation of the i-th grid and the plane indicated by the local equation of the adjacent preceding or following grid.

[0071] By adjusting the parameters of each local equation, the sum of the first distance sum, the second distance sum, and the included angle sum is made smaller, thus ensuring that a better local equation is obtained.

[0072] In the third approach, for cases with high-precision maps, the ground equations can be obtained based on the high-precision maps. This approach can obtain both global and local equations within the ground equations.

[0073] The global equations can be obtained through the following process:

[0074] Process A3: Obtain the points corresponding to the road surface area from the high-precision map.

[0075] Process B3: Determine the ground equation of the road surface area in the coordinate system of the acquisition device based on the coordinates of the points corresponding to the road surface area.

[0076] For example, the distance from each point to the plane indicated by the initialized global equation is calculated based on the coordinates of the points corresponding to the road surface area. The parameters of the global equation are then optimized based on the distances of each point to make the distances of each point smaller, thereby obtaining a better global equation.

[0077] The optimization process for local equations is similar to that for global equations. The difference is that the points used to calculate the distance are the points of the plane matching corresponding to the local equation. Unrelated points in this plane do not participate in the distance calculation, so it will not be elaborated further.

[0078] Sub-step S2062: Based on the ground equation, determine the first spatial coordinates of each first pixel in the road surface area within the coordinate system of the acquisition device.

[0079] In this embodiment, for cases where the ground equation includes a global equation covering the road surface area, sub-step S2062 can be implemented through the following process:

[0080] Process A4: Determine the position of each of the first pixels in the imaging plane of the acquisition device.

[0081] Since the road image and the imaging plane correspond, the position of the first pixel within the imaging plane of the acquisition device can be determined based on its position within the road surface area. For example, as... Figure 4A As shown, point P on the imaging plane is the location of a first pixel within the road surface area.

[0082] Process B4: Based on the position of the first pixel on the imaging plane, the optical center of the acquisition device, and the global equation, calculate the coordinates of the intersection point between the imaging ray corresponding to the first pixel and the global plane indicated by the global equation.

[0083] The optical center of the acquisition device is like Figure 4A Midpoint O. The imaging ray is the line connecting the first pixel and the optical center, that is, the line connecting point O and point P is the imaging ray of the first pixel.

[0084] The coordinates of the intersection point of the imaging ray and the global plane indicated by the global equation can be obtained based on the principle of line-plane intersection.

[0085] In a specific example, let the position of the first pixel in the imaging plane be (u, v). Then, the coordinates are normalized as follows: X = (u - cx) / fx, Y = (v - cy) / fy, z = 1. Here, fx and fy are the focal lengths of the acquisition device (i.e., the camera), and cx and cy are the principal points of the acquisition device (i.e., the camera). These four parameters are all intrinsic parameters of the acquisition device (i.e., the camera).

[0086] The depth d is recovered from the ground equation: A·(dx)+B·(dy)+C·(dz)+D=0, d=-D / (Ax+By+C).

[0087] The coordinates of the intersection point in the coordinate system of the acquisition device can be represented as (dx, dy, d).

[0088] Process D4: Determine the first spatial coordinates of the first pixel in the coordinate system of the acquisition device based on the coordinates of the intersection point with the global plane.

[0089] In one case, if the ground equation does not contain local equations, the coordinates of the intersection point can be directly used as the first spatial coordinates of the first pixel.

[0090] In another case, such as Figure 4B As shown, if the ground equation includes a local equation covering part of the road surface area, then process D4 can be implemented as follows: obtain the local equation corresponding to the first pixel based on the coordinates of the intersection point with the global plane; calculate the coordinates of the intersection point of the imaging ray and the local plane indicated by the local equation, and use it as the first spatial coordinates of the first pixel.

[0091] In the presence of local equations, to improve accuracy, the first spatial coordinates of the first pixel can be based on the coordinates of the intersection point between the calculated imaging ray and the local plane indicated by the local equation. The calculation method for the coordinates of the intersection point with the local plane is similar to that for the intersection point with the global plane, and therefore will not be repeated.

[0092] Step S208: Based on the pose of the trajectory point corresponding to the road surface area to which the first pixel belongs, determine the second spatial coordinates of the mapping point in the world coordinate system to which the first pixel is mapped, so as to generate a point cloud map based on the second spatial coordinates.

[0093] In one feasible approach, the second spatial coordinates are denoted as (xword, yword, zword) = T·(dx, dy, d), where T is the pose of the trajectory corresponding to the road surface region to which the first pixel belongs.

[0094] Optionally, such as Figure 5 As shown, the method also includes the following steps:

[0095] Step S210: Obtain the color information corresponding to the first pixel.

[0096] For example, the RGB value corresponding to the first pixel is extracted from the road surface area to which it belongs as color information.

[0097] Step S212: Project the points in the point cloud map under the world coordinate system according to the height direction to obtain the second pixel of the first pixel in the two-dimensional map.

[0098] In one feasible approach, such as Figure 6 As shown, step S212 can be implemented through the following sub-steps:

[0099] Sub-step S2121: Segment the point cloud in the point cloud map according to distance.

[0100] The distance between segments can be determined as needed and is not restricted.

[0101] Sub-step S2122: For each point cloud segment cut out, determine the upper limit set and lower limit set of the point cloud segment based on the second spatial coordinates of each point within the point cloud segment.

[0102] The upper limit set includes: longitude upper limit (denoted as Bmax), latitude upper limit (Lmax), and altitude upper limit (Hmax), and the lower limit set includes longitude lower limit (Bmin), latitude lower limit (Lmin), and altitude lower limit (Hmin).

[0103] The upper limit for longitude is the longitude of the point cloud segment with the largest longitude. The upper limit for latitude is the latitude of the point cloud segment with the largest latitude. The upper limit for altitude is the altitude of the point cloud segment with the largest altitude.

[0104] The lower limit for longitude is the longitude of the point with the smallest longitude in the point cloud segment, and the lower limit for latitude is the latitude of the point with the smallest latitude in the point cloud segment. The lower limit for altitude is the altitude of the point with the smallest altitude in the point cloud segment.

[0105] Sub-step S2123: Use the lower bound set of each point cloud segment as the origin coordinates of the point cloud segment, and obtain the resolution of the point cloud segment.

[0106] In this embodiment, the origin coordinates of the point cloud segment can be represented as (B min ,L min H min ).

[0107] The resolution corresponding to point cloud segments can be denoted as (resolution). B resolution L resolution H ), where resolution B This indicates the number of pixels per meter of longitude. L This indicates the number of pixels per meter in latitude. H This indicates the number of pixels corresponding to 1 meter in height.

[0108] Sub-step S2124: Based on the origin coordinates and resolution of the point cloud segment, project each point in the point cloud segment along the height direction to obtain a two-dimensional map fragment.

[0109] Sub-step S2125: The two-dimensional map segments of the point cloud are stitched together according to the upper limit set and lower limit set of the point cloud segments to obtain a two-dimensional map.

[0110] This two-dimensional map is a stitch-together view of the road surface portion from various road images.

[0111] Step S214: Determine the color information of the second pixel based on the color information of the first pixel.

[0112] By assigning the color information of the first pixel to the color information of the corresponding second pixel, the stitched 2D map can display the actual road surface conditions. The stitched result is as follows: Figure 7 As shown. This allows for convenient observation of the road surface condition, and image recognition of the 2D map can determine whether cracks exist in the road surface, thus improving inspection speed. Among other things, Figure 7 The left side of the image is a sequence of road images, while the right side can be a stitched-together two-dimensional map.

[0113] It should be noted that, in addition to generating a 2D map from a point cloud map, a height map can also be generated from the point cloud map. The longitude, latitude, and altitude of each second pixel can also be calculated from the 2D map. Alternatively, a target object can be directly selected from the 2D map for positioning and measurement. The longitude, latitude, and altitude of the second pixel can be determined in the following way:

[0114] Determine the position of the selected second pixel on the 2D map (denoted as (index)). X ,index Y And obtain the index of the second pixel in the height direction on the height map (denoted as index). H Based on the calculated origin coordinates (B) of the point cloud segments. min ,L min H min ) and resolution of 2D maps B resolution L resolution H The formulas for calculating latitude, longitude, and altitude are as follows:

[0115] B = B min +index x *resolution B .

[0116] L = L min +index Y *resolution L .

[0117] H = H min +index H *resolution H .

[0118] This method can construct road surface maps based on road images, which consist of a point cloud map and a two-dimensional map (i.e., a two-dimensional photo map). It can perform coarse and fine calculations of the spatial coordinates of the first pixel of the road surface, providing a method for depth estimation using monocular acquisition devices. Furthermore, it can accurately obtain the geographical location of each second pixel based on the two-dimensional road surface map, enabling target localization and measurement from the two-dimensional map. The two-dimensional map includes rich colors, thus compensating for the poor reflectivity of the point cloud in LiDAR.

[0119] Example 2

[0120] Reference Figure 8 The diagram shows a structural block diagram of the apparatus according to Embodiment 2 of this application.

[0121] The device includes:

[0122] The first acquisition module 802 is used to acquire a sequence of road images and a sequence of trajectory points of the acquisition device for the road images, wherein the trajectory points are used to indicate the pose of the acquisition device in the world coordinate system when acquiring the corresponding road images;

[0123] The second acquisition module 804 is used to acquire the road surface area from the road image;

[0124] The third acquisition module 806 is used to acquire the first spatial coordinates of the first pixel of the road surface area in the coordinate system of the acquisition device.

[0125] The first determining module 808 is used to determine the second spatial coordinates of the mapping point of the first pixel in the world coordinate system based on the pose of the trajectory point corresponding to the road surface area to which the first pixel belongs, so as to generate a point cloud map based on the second spatial coordinates.

[0126] Optionally, the second acquisition module 804 is used to identify road edges from the road image; and to extract the road surface area from the road image based on the identified road edges.

[0127] Optionally, the third acquisition module 806 is used to acquire the ground equation of the road surface area in the coordinate system of the acquisition device; and determine the first spatial coordinates of each first pixel in the road surface area in the coordinate system of the acquisition device based on the ground equation.

[0128] Optionally, the ground equation includes at least a global equation covering the road surface area; the third acquisition module 806 is used to determine the position of each first pixel in the imaging plane of the acquisition device; based on the position of the first pixel in the imaging plane, the optical center of the acquisition device, and the global equation, calculate the coordinates of the intersection point of the imaging ray corresponding to the first pixel and the global plane indicated by the global equation, wherein the imaging ray is the line connecting the first pixel and the optical center; and determine the first spatial coordinates of the first pixel in the coordinate system of the acquisition device based on the coordinates of the intersection point with the global plane.

[0129] Optionally, if the ground equation includes a local equation covering a portion of the road surface area, the third acquisition module 806 is used to acquire the local equation corresponding to the first pixel based on the coordinates of the intersection point with the global plane; and calculate the coordinates of the intersection point of the imaging ray and the local plane indicated by the local equation as the first spatial coordinates of the first pixel.

[0130] Optionally, the third acquisition module 806 is used to identify feature points from the road image; associate feature points in two adjacent road images with corresponding points to obtain corresponding point pairs, the corresponding point pairs including two corresponding points; determine the residual of the two corresponding points in the corresponding point pair in the world coordinate system according to the initialized ground equation and the pose of the trajectory points of the acquisition device; and optimize at least the initialized ground equation according to the residual to obtain the optimized ground equation as a global equation covering the road surface area.

[0131] Optionally, the third acquisition module 806 is used to acquire the laser point cloud collected by the lidar; acquire the ground point cloud from the laser point cloud; perform grid processing on the road surface covered by the ground point cloud and initialize the local equations corresponding to the grids; and optimize the local equations of each grid according to the poses of the ground points contained in each grid in the grid sequence to obtain the optimized local equations of the grids.

[0132] Optionally, the third acquisition module 806 is used to calculate, based on the pose of the ground points contained in each grid and the local equation of each grid, a first distance from each ground point to the ground of its respective grid, a second distance between the grounds of two adjacent grids, and an angle between the grounds of two adjacent grids, wherein the ground is determined according to the local equation; and to optimize the local equation of each grid to obtain the local equation of the grid that minimizes the sum of the first distance, the sum of the second distance, and the sum of the angle corresponding to each grid.

[0133] Optionally, the third acquisition module 806 is used to acquire points corresponding to the road surface area from a high-precision map; and to determine the ground equation of the road surface area in the coordinate system of the acquisition device based on the coordinates of the points corresponding to the road surface area.

[0134] Optionally, the device further includes:

[0135] The fourth acquisition module 810 is used to acquire the color information corresponding to the first pixel;

[0136] The fifth acquisition module 812 is used to project the points in the point cloud map under the world coordinate system according to the height direction to obtain the second pixel of the first pixel in the two-dimensional map;

[0137] The second determining module 814 is used to determine the color information of the second pixel based on the color information of the first pixel.

[0138] Optionally, the fifth acquisition module 812 is used to segment the point cloud in the point cloud map according to distance; for each segmented point cloud segment, based on the second spatial coordinates of each point within the point cloud segment, determine the upper limit set and lower limit set of the point cloud segment, wherein the upper limit set includes: upper limit value of longitude, upper limit value of latitude, and upper limit value of altitude, and the lower limit set includes lower limit value of longitude, lower limit value of latitude, and lower limit value of altitude; use the lower limit set of each point cloud segment as the origin coordinates of the point cloud segment, and obtain the resolution of the point cloud segment; based on the origin coordinates and resolution of the point cloud segment, project each point within the point cloud segment along the height direction to obtain a two-dimensional map fragment; stitch the two-dimensional map fragments of the point cloud segment according to the upper limit set and lower limit set of the point cloud segment to obtain a two-dimensional map.

[0139] The device can achieve the same effect as the aforementioned method, so it will not be described in detail here.

[0140] Example 3

[0141] Reference Figure 9 The diagram shows a structural schematic of an electronic device according to Embodiment 3 of this application. The specific embodiments of this application do not limit the specific implementation of the electronic device.

[0142] like Figure 9 As shown, the electronic device may include: a processor 902, a communications interface 904, a memory 906, and a communications bus 908.

[0143] in:

[0144] The processor 902, communication interface 904, and memory 906 communicate with each other via communication bus 908.

[0145] Communication interface 904 is used to communicate with other electronic devices or servers.

[0146] The processor 902 is used to execute program 910, specifically the relevant steps in the above method embodiments.

[0147] Specifically, program 910 may include program code that includes computer operation instructions.

[0148] The processor 902 may be a CPU, an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application. The smart device includes one or more processors, which may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0149] Memory 906 is used to store program 910. Memory 906 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0150] Specifically, program 910 can be used to cause processor 902 to perform the operation corresponding to the method described in any of the foregoing multiple method embodiments.

[0151] The specific implementation of each step in program 910 can be found in the corresponding descriptions of the steps and units in the above method embodiments, and has corresponding beneficial effects, which will not be repeated here. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding process descriptions in the foregoing method embodiments, and will not be repeated here.

[0152] This application also provides a computer program product, including computer instructions that instruct a computing device to perform an operation corresponding to any of the methods in the above-described multiple method embodiments.

[0153] It should be noted that, depending on the implementation needs, the various components / steps described in the embodiments of this application can be broken down into more components / steps, or two or more components / steps or parts of the operation of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of this application.

[0154] The methods described in the embodiments of this application can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code downloaded over a network that is originally stored in a remote recording medium or a non-transitory machine-readable medium and will be stored in a local recording medium. Thus, the methods described herein can be processed by software stored on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an ASIC or FPGA). It is understood that the computer, processor, microprocessor controller, or programmable hardware includes storage components (e.g., RAM, ROM, flash memory, etc.) capable of storing or receiving software or computer code that, when accessed and executed by the computer, processor, or hardware, implements the methods described herein. Furthermore, when a general-purpose computer accesses code used to implement the methods shown herein, the execution of the code transforms the general-purpose computer into a dedicated computer for executing the methods shown herein.

[0155] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this application.

[0156] The above embodiments are only used to illustrate the embodiments of this application, and are not intended to limit the embodiments of this application. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of this application. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of this application, and the patent protection scope of the embodiments of this application should be defined by the claims.

Claims

1. A map generation method, comprising: Acquire a sequence of road images and a sequence of trajectory points of the road image acquisition device, wherein the trajectory points are used to indicate the pose of the acquisition device in the world coordinate system when acquiring the corresponding road image; Identify the road edges from the road images; Based on the identified road edges, the road surface area is extracted from the road image; Obtain the ground equation of the road surface area in the coordinate system of the acquisition device; Based on the ground equation, determine the first spatial coordinates of each first pixel in the road surface area within the coordinate system of the acquisition device; Based on the pose of the trajectory point corresponding to the road surface area to which the first pixel belongs, the second spatial coordinates of the mapping point in the world coordinate system are determined from the first spatial coordinates of the first pixel. A two-dimensional map for determining whether there are cracks in the road surface is generated from the point cloud map generated based on the second spatial coordinates. The step of obtaining the ground equation of the road surface area in the coordinate system of the acquisition device includes: Identify feature points from the road image; Feature points in two adjacent road images are associated with corresponding points to obtain pairs of corresponding points, each pair of corresponding points consisting of two corresponding points. Based on the initialized ground equations and the pose of the trajectory points of the acquisition device, determine the residuals of the two corresponding points in the world coordinate system. Based on the residual, at least the initialized ground equation is optimized to obtain the optimized ground equation as a global equation covering the road surface area.

2. The method according to claim 1, wherein, The ground equations include at least global equations covering the road surface area; Determining the first spatial coordinates of each first pixel in the road surface region within the coordinate system of the acquisition device based on the ground equation includes: Determine the position of each first pixel in the imaging plane of the acquisition device; Based on the position of the first pixel on the imaging plane, the optical center of the acquisition device, and the global equation, calculate the coordinates of the intersection point of the imaging ray corresponding to the first pixel and the global plane indicated by the global equation, where the imaging ray is the line connecting the first pixel and the optical center. The first spatial coordinates of the first pixel in the coordinate system of the acquisition device are determined based on the coordinates of the intersection point with the global plane.

3. The method according to claim 2, wherein, If the ground equation includes a local equation covering a portion of the road surface area, then the first spatial coordinates of the first pixel in the coordinate system of the acquisition device are determined based on the coordinates of the intersection point with the global plane, including: Based on the coordinates of the intersection point with the global plane, obtain the local equation corresponding to the first pixel; The coordinates of the intersection point of the imaging ray and the local plane indicated by the local equation are calculated and used as the first spatial coordinates of the first pixel.

4. The method according to claim 1, wherein, The process of obtaining the ground equation of the road surface area within the coordinate system of the acquisition device includes: Acquire laser point clouds collected by lidar; Obtain ground point cloud from the laser point cloud; The road surface covered by the ground point cloud is rasterized, and the local equations corresponding to the raster are initialized. Based on the pose of the ground points contained in each grid in the grid sequence, the local equations of each grid are optimized to obtain optimized local equations of the grid.

5. The method according to claim 4, wherein, The step of optimizing the local equations of each grid cell based on the poses of the ground points contained in each grid cell in the grid sequence to obtain optimized local equations for the grid cells includes: Based on the pose of the ground points contained in each grid and the local equation of each grid, calculate the first distance from each ground point to the ground of its grid, the second distance between the grounds of two adjacent grids, and the angle between the grounds of two adjacent grids, wherein the ground is determined according to the local equation; The local equations of each grid are optimized to obtain the local equation that minimizes the sum of the first distance, the sum of the second distance, and the sum of the included angles corresponding to each grid.

6. The method according to claim 1, wherein, The process of obtaining the ground equation of the road surface area within the coordinate system of the acquisition device includes: Obtain the points corresponding to the road surface area from the high-precision map; Based on the coordinates of the points corresponding to the road surface area, the ground equation of the road surface area in the coordinate system of the acquisition device is determined.

7. The method according to claim 1, wherein, The method further includes: Obtain the color information corresponding to the first pixel; The points in the point cloud map in the world coordinate system are projected according to the height direction to obtain the second pixel of the first pixel in the two-dimensional map. The color information of the second pixel is determined based on the color information of the first pixel.

8. The method according to claim 7, wherein, The step of projecting points in the point cloud map under the world coordinate system according to the height direction to obtain the second pixel of the first pixel in the two-dimensional map includes: The point cloud in the point cloud map is segmented according to distance; For each segmented point cloud, the upper limit set and lower limit set of the point cloud segment are determined based on the second spatial coordinates of each point within the point cloud segment. The upper limit set includes: upper limit value of longitude, upper limit value of latitude, and upper limit value of altitude. The lower limit set includes lower limit value of longitude, lower limit value of latitude, and lower limit value of altitude. The lower bound set of each point cloud segment is used as the origin coordinate of the point cloud segment, and the resolution of the point cloud segment is obtained. Based on the origin coordinates and resolution of the point cloud segment, each point in the point cloud segment is projected along the height direction to obtain a two-dimensional map fragment. The two-dimensional map segments of the point cloud are stitched together according to the upper and lower bound sets of the point cloud segments to obtain a two-dimensional map.

9. A map generation apparatus, comprising: The first acquisition module is used to acquire a sequence of road images and a sequence of trajectory points of the acquisition device for the road images, wherein the trajectory points are used to indicate the pose of the acquisition device in the world coordinate system when acquiring the corresponding road images; The second acquisition module is used to identify road edges from the road image; and to extract the road surface region from the road image based on the identified road edges. The third acquisition module is used to acquire the ground equation of the road surface area in the coordinate system of the acquisition device; and to determine the first spatial coordinates of each first pixel in the road surface area in the coordinate system of the acquisition device based on the ground equation. The first determining module is used to determine the second spatial coordinates of the mapping point in the world coordinate system from the first spatial coordinates of the first pixel to the pose of the trajectory point corresponding to the road surface area to which the first pixel belongs, so as to generate a two-dimensional map for determining whether there are cracks in the road surface based on the point cloud map generated according to the second spatial coordinates. The step of obtaining the ground equation of the road surface area in the coordinate system of the acquisition device includes: Identify feature points from the road image; Feature points in two adjacent road images are associated with corresponding points to obtain pairs of corresponding points, each pair of corresponding points consisting of two corresponding points. Based on the initialized ground equations and the pose of the trajectory points of the acquisition device, determine the residuals of the two corresponding points in the world coordinate system. Based on the residual, at least the initialized ground equation is optimized to obtain the optimized ground equation as a global equation covering the road surface area.

10. An electronic device, comprising: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction that causes the processor to perform the operation corresponding to the method as described in any one of claims 1-8.

11. A computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any one of claims 1-8.

Images