Map feature generation method and computer program product

Abstract

Embodiments of the present application provide a map element generation method and a computer program product, wherein the map element generation method comprises: identifying an image to obtain image coordinates of shape points of a map element recorded by the image; performing visual three-dimensional reconstruction based on the image coordinates of the shape points to obtain initial geographic coordinates of corresponding position points of the shape points in an actual physical space; determining a neighborhood of the position points in a laser point cloud corresponding to the image according to the initial geographic coordinates of the position points; fitting a function expression representing a shape of the map element based on laser points in the neighborhood; determining a projection error of the shape points based on the function expression and the image coordinates of the shape points; performing nonlinear optimization based on the projection error and a set laser geometric constraint parameter, and determining geographic coordinates of the position points of the map element according to a result of the nonlinear optimization. The advantages of the image and the laser point cloud are combined, and the precision and the update rate of the electronic map are improved under the premise of low cost.

Description

Technical Field

[0001] This application relates to the field of geographic information technology, and in particular to a map feature generation method and computer program product. Background Technology

[0002] With the development of network technology, the application scenarios of electronic maps are becoming increasingly widespread. For example, users can use map navigation applications based on electronic map data to access services such as navigation, positioning, and location search. To meet the needs of scenarios such as intelligent driving and smart cities, electronic maps are evolving from standard maps to high-precision maps. Compared with standard maps, high-precision maps provide a richer representation of the real world and have higher accuracy. Related technologies use high-precision acquisition equipment integrating high-level LiDAR to collect production data for creating high-precision maps. However, high-precision acquisition equipment is very expensive, and the real world is constantly changing. If high-precision maps need to be updated efficiently to accurately represent the real world, large-scale deployment of high-precision acquisition equipment is required, which leads to excessively high update costs. To address this, related technologies propose replacing the high-level LiDAR in high-precision acquisition equipment with low-cost LiDAR or cameras. While this approach reduces costs, the production data collected by low-cost LiDAR or cameras has various problems, resulting in limited accuracy of high-precision maps created based on such data, which cannot meet the needs of real-world scenarios. Summary of the Invention

[0003] In view of this, embodiments of this application provide a map feature generation method and a computer program product to at least partially solve the above problems.

[0004] According to a first aspect of the embodiments of this application, a map feature generation method is provided, comprising: recognizing an image to obtain image coordinates of shape points of map features recorded in the image; performing visual three-dimensional reconstruction based on the image coordinates of the shape points to obtain initial geographic coordinates of the location points corresponding to the shape points in actual physical space; determining the neighborhood of the location points in the laser point cloud corresponding to the image based on the initial geographic coordinates of the location points; fitting a function expression characterizing the shape of the map feature based on the laser points in the neighborhood; determining the projection error of the shape points based on the function expression and the image coordinates of the shape points; performing nonlinear optimization based on the projection error and set laser geometric constraint parameters, and determining the geographic coordinates of the location points of the map features based on the result of the nonlinear optimization.

[0005] According to a second aspect of the embodiments of this application, a map feature generation apparatus is provided, comprising: an identification module for identifying an image to obtain image coordinates of shape points of map features recorded in the image; a reconstruction module for performing visual three-dimensional reconstruction based on the image coordinates of the shape points to obtain the initial geographic coordinates of the corresponding location points of the shape points in actual physical space; a laser point cloud module for determining the neighborhood of the location points in the laser point cloud corresponding to the image based on the initial geographic coordinates of the location points; a function module for fitting a function expression characterizing the shape of the map feature based on the laser points in the neighborhood; an error module for determining the projection error of the shape points based on the function expression and the image coordinates of the shape points; and a nonlinear optimization module for performing nonlinear optimization based on the projection error and set laser geometric constraint parameters, and determining the geographic coordinates of the location points of the map features based on the result of the nonlinear optimization.

[0006] According to a third aspect of the embodiments of this application, an electronic device is provided, including: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other through the communication bus; the memory is used to store at least one executable instruction, which causes the processor to perform an operation corresponding to the map feature generation method of the first aspect.

[0007] According to a fourth aspect of the embodiments of this application, a storage medium is provided that stores a computer program thereon, which, when executed by a processor, implements the map feature generation method of the first aspect.

[0008] According to a fifth aspect of the embodiments of this application, a computer program product is provided that, when executed by a processor, implements the map feature generation method as described in the first aspect.

[0009] The map feature generation method and computer program product provided in this application identify an image to obtain the image coordinates of the shape points of the map features recorded in the image; based on the image coordinates of the shape points, perform visual 3D reconstruction to obtain the initial geographic coordinates of the corresponding location points in the actual physical space; determine the neighborhood of the location points in the laser point cloud corresponding to the image based on the initial geographic coordinates of the location points; fit a function expression representing the shape of the map feature based on the laser points in the neighborhood; determine the projection error of the shape points based on the function expression and the image coordinates of the shape points; perform nonlinear optimization based on the projection error and set laser geometric constraint parameters, and determine the geographic coordinates of the location points of the map features based on the result of the nonlinear optimization. By combining the image coordinates of the shape points in the image and the projection error determined by projecting using the function expression fitted by the laser point cloud, nonlinear optimization is performed. By combining the accuracy of the image captured by the camera and the accuracy of the laser point cloud, the accuracy and update rate of the electronic map are improved while maintaining a low cost. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0011] Figure 1 This is a schematic diagram of a map feature generation method provided in Embodiment 1 of this application;

[0012] Figure 2 A flowchart of a map feature generation method provided in Embodiment 1 of this application;

[0013] Figure 3 This is a schematic diagram of a projection principle provided in Embodiment 1 of this application;

[0014] Figure 4 This is a schematic diagram of another projection principle provided in Embodiment 1 of this application;

[0015] Figure 5 This is another schematic diagram of the projection principle provided in Embodiment 1 of this application;

[0016] Figure 6 This is a structural diagram of a map feature generation device provided in Embodiment 2 of this application;

[0017] Figure 7 This is a structural diagram of an electronic device provided in Embodiment 3 of this application. Detailed Implementation

[0018] To enable those skilled in the art to better understand the technical solutions in the embodiments of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art should fall within the protection scope of the embodiments of this application.

[0019] The specific implementation of the embodiments of this application will be further described below with reference to the accompanying drawings.

[0020] Example 1

[0021] This application provides a map feature generation method in an electronic device. For ease of understanding, the application scenario of the map feature generation method provided in this application is described below. (Refer to...) Figure 1 As shown, Figure 1This is a schematic diagram of a map feature generation method provided in Embodiment 1 of this application. Figure 1 The scenario shown includes an electronic device 101, a data acquisition device 102, and map elements 103. Map elements 103 are objects in actual physical space; for example, a lane line, turn arrow, road boundary, artificial pole, traffic sign, traffic light, guardrail, etc. The electronic device 101 can be a device that executes the map element generation method provided in Embodiment 1 of this application. In some optional application scenarios, the map element generation method provided in this embodiment can create high-precision maps, improving the accuracy and update rate of high-precision maps.

[0022] Electronic device 101 can be a terminal device such as a smartphone, tablet computer, laptop computer, or vehicle terminal. Electronic device 101 can also be a network device such as a server. Of course, this is only an illustrative example and does not mean that this application is limited to this.

[0023] The acquisition device 102 can be used to acquire images captured by a camera and laser point clouds detected by a lidar. The acquisition device 102 can transmit the images captured by the camera and the laser point clouds detected by the lidar to the cloud, and then send them to the electronic device 101 via the cloud. Alternatively, it can directly send the images captured by the camera and the laser point clouds detected by the lidar to the electronic device 101.

[0024] Electronic device 101 can access a network, connect to the cloud via the network, and exchange data. In this application, the network includes a Local Area Network (LAN), a Wide Area Network (WAN), and mobile communication networks, such as the World Wide Web (WWW), Long Term Evolution (LTE) networks, 2G networks, 3G networks, and 5G networks. The cloud can include various devices connected via the network, such as servers, relay devices, and device-to-device (D2D) devices. Of course, this is only an illustrative example and does not mean that this application is limited to this.

[0025] Combination Figure 1 The scenario shown illustrates the map feature generation method provided in Embodiment 1 of this application. It should be noted that... Figure 1This is merely one application scenario of the map feature generation method provided in Embodiment 1 of this application, and does not imply that the map feature generation method must be applied to all map feature generation methods. Figure 1 Optionally, the map feature generation method provided in this application embodiment can be applied to an electronic device, that is, the electronic device is the executing entity of the map feature generation method provided in this application embodiment. Figure 2 As shown, Figure 2 A flowchart of a map feature generation method provided in Embodiment 1 of this application is provided. The method includes the following steps:

[0026] Step 201: Recognize the image to obtain the image coordinates of the shape points of the map features recorded in the image.

[0027] Map elements can be any element in an image; for example, a map element can be an object or a sign. Furthermore, map elements can be elements used to create electronic maps, that is, elements that need to be displayed on the electronic map; for example, map elements can be ground, lane lines, turn arrows, road boundaries, artificial poles, traffic signs, traffic lights, guardrails, etc. The shape point of a map element can be any point within the map elements in the image.

[0028] It should be noted that the coordinates in this application refer to coordinates in the camera coordinate system, or coordinates in the actual physical space coordinate system. Since coordinate transformations are possible between different coordinate systems, this application does not restrict which coordinate system is used; multiple coordinate systems can be used, or only one coordinate system can be used. Specifically, the camera coordinate system refers to a coordinate system established with the camera as the reference point. For example, the camera coordinate system could be a coordinate system with the optical center of the camera lens as the origin. The actual physical space coordinate system refers to a coordinate system established with the actual physical space, i.e., the real world, as the reference point.

[0029] For example, the image coordinates of the shape point can be the coordinates of the shape point in the camera coordinate system. Recognizing the image to obtain the image coordinates of the shape points of the map features recorded in the image includes: recognizing the image, obtaining the pixel positions of the shape points of the map features in the image, and determining the image coordinates of the shape points based on the transformation relationship between the pixel positions and the camera coordinate system. Because the image is two-dimensional, the pixel position can be represented as (x, y), that is, the shape point is the pixel in the x-th row and y-th column of the image. Of course, this is just an example.

[0030] Step 202: Based on the image coordinates of the shape point, perform visual 3D reconstruction to obtain the initial geographic coordinates of the corresponding location point of the shape point in the actual physical space.

[0031] It should be noted that shape points are the points in an image captured by a camera that are imaged on the imaging plane. Position points and shape points correspond to object points and image points, respectively. Visual 3D reconstruction can be performed based on the principles of photographic geometry, which include the assumption that the optical center, object point, and image point lie on a straight line. Based on this principle, for the same object point, multiple image points can be obtained by capturing images from different angles, and the extensions of the lines connecting each image point to the optical center all pass through the object point. Here, two specific visual 3D reconstruction methods are illustrated.

[0032] Optionally, in the first approach, the map element in this application is a ground element. Based on the image coordinates of the shape point, visual 3D reconstruction is performed to obtain the initial geographic coordinates of the shape point's corresponding location in the actual physical space. This includes: calculating the coordinates of the intersection point of the ray passing from the optical center through the shape point and the preset surface based on the image coordinates of the shape point and the initial function expression of the preset surface, which serves as the initial geographic coordinates of the shape point's corresponding location in the actual physical space. Here, the optical center is the center of the camera lens, which can be the origin of the camera coordinate system, and the location point lies within the preset surface. The preset surface is the ground, which can be a plane or a curved surface; for example, the preset surface can be a ground plane or the surface of the ground. It should be noted that if the map element is a sign surface, the preset surface can also be the plane of the sign. Figure 3 As shown, taking the ground as the preset surface as an example, Figure 3 In the diagram, the point P where the ray from the optical center (i.e. the center of the camera lens) C passes through the shape point X1 and intersects the ground is the position point. The ground can be represented as ax + by + cz + d = 0. Of course, this is just an example.

[0033] Optionally, in the second approach, the map element is a sign element or a pole element. Based on the image coordinates of the shape points, visual 3D reconstruction is performed to obtain the initial geographic coordinates of the shape points' corresponding locations in the actual physical space. This includes: using the image coordinates of the shape points in the first image as a ray extending outward from the shape points in the first image through the first optical center, and using the image coordinates of the shape points in the second image as a ray extending outward from the shape points in the second image through the second optical center, the coordinates of the intersection of the first and second rays are determined as the initial geographic coordinates of the shape points' corresponding locations in the actual physical space. Optionally, the first image is the image of the sign element observed in the first frame, and the second image is the image of the sign element or pole element observed in the last frame. The first and second images can be images taken from different angles. Because of the different angles, the positions of the shape points corresponding to the same location points are different in the first and second images. Figure 4As shown, for the same location point P, the corresponding shape points in the first image and the second image are X1 and X2, respectively. For the first image, a ray from the optical center C1 that passes through the shape point X1 of the first image will pass through location point P; for the second image, a ray from the optical center C2 that passes through the shape point X2 of the second image will also pass through location point P. Therefore, the extensions of C1X1 and C2X2 are the location point P. Figure 4 In the diagram, for ease of understanding, C1X1 and C2X2 form an obtuse angle. In practical applications, however, they can form either an obtuse or an acute angle. For example, if the camera lens only points in one direction and cannot rotate, then C1X1 and C2X2 can be an acute angle; conversely, if the camera lens can rotate and can shoot forward or backward, then C1X1 and C2X2 can be an obtuse angle. Of course, the above are merely illustrative examples. Optionally, taking a rod element as an example, if the map element is a rod element, then the shape points include: the shape points corresponding to the vertices and the shape points corresponding to the bases of the rod.

[0034] Step 203: Determine the neighborhood of the location point in the corresponding laser point cloud in the image based on the initial geographic coordinates of the location point.

[0035] It should be noted that a laser point cloud refers to the set of points obtained after acquiring the spatial coordinates of each sampling point of an object using laser light. A laser point cloud can include the three-dimensional coordinates of each point, as well as laser reflection intensity and color information (RGB). It should also be noted that in this application, the neighborhood of a location point refers to the set of points whose distance from the location point is less than or equal to a preset distance. The preset distance can be set by the user; it can be greater than 0, for example, less than or equal to 10m. The preset distance could be 1m, 2m, 3m, 5m, etc., and this is merely an example.

[0036] Step 204: Fit a function expression representing the shape of map features based on laser points in the neighborhood.

[0037] The map element can be a preset surface, such as a preset plane or a preset curved surface, or a preset line segment; this is merely an example. It should be noted that the map element is imaged in the image during camera capture. For example, the map element can be the ground (i.e., the ground feature). During capture, the ground is imaged in the image, and a function expression for the ground can be obtained by fitting the ground to the laser point cloud. Points that satisfy this function expression are the location points on the ground. This is just an example.

[0038] Optionally, in one implementation, the initial geographic coordinates of a location point in the actual physical space can be re-determined based on the function expression of the map feature and the image coordinates of the shape point. A specific example is provided here for illustration. Optionally, in one specific implementation, re-determining the initial geographic coordinates of a location point in the actual physical space based on the function expression of the map feature and the image coordinates of the shape point includes: calculating the coordinates of the intersection point of the ray passing through the shape point from the optical center and the map feature, using this as the fitted coordinates of the location point. The initial geographic coordinates of the location point re-determined using the function expression of the map feature are based on laser point clouds, taking into account the accuracy of the LiDAR, which helps to improve accuracy.

[0039] Step 205: Determine the projection error of the shape points based on the function expression and the image coordinates of the shape points.

[0040] It should be noted that projection error is used to indicate the error between the image coordinates and projected coordinates of a shape point. Optionally, the projection error of a shape point is determined based on a function expression and the image coordinates of the shape point, including: calculating the coordinates of the intersection point of the line passing through the position point and the optical center with the imaging plane based on the fitted coordinates of the position point determined using the function expression, and using this as the projected coordinates of the shape point; and calculating the sum of the errors between the image coordinates and projected coordinates of the shape point in at least one image as the projection error. The fitted coordinates of the position point are coordinates determined based on the function expression. Because the function expression is based on fitting laser points within the initial geographic coordinate neighborhood of the position point, the fitted coordinates of the position point depend on the accuracy of the laser point cloud. The projection error between the projected coordinates of the shape point determined by the fitted coordinates of the position point and the image coordinates of the shape point can also indicate the error between the camera-captured image and the laser point cloud image.

[0041] Step 206: Perform nonlinear optimization based on projection error and set laser geometric constraint parameters, and determine the geographic coordinates of the location points of map elements based on the results of nonlinear optimization.

[0042] It should be noted that projection error indicates the error between the image coordinates and projected coordinates of a shape point, while laser geometric constraint parameters indicate the error between the initial geographic coordinates and the fitted coordinates of a location point. The nonlinear optimization process can move the location point, or transform its fitted coordinates. The error is calculated using the fitted coordinates of the location point and the image coordinates of the shape point. The fitted coordinates of the location point, where the error (including projection error and laser geometric constraint parameters) is minimized, are determined as the geographic coordinates of the location point. Since the image coordinates of the shape point are determined from camera images and the fitted coordinates of the location point are determined from laser point clouds detected by LiDAR, the geographic coordinates of the location point determined when the error between the two is minimized comprehensively consider the accuracy of both the camera and LiDAR, improving mapping accuracy while maintaining low cost. Optionally, for at least one image, steps 201-204 can be performed on each image to obtain the image coordinates of at least one shape point and the fitted coordinates of at least one location point. Calculating the error using the image coordinates of at least one shape point and the fitted coordinates of at least one location point reduces the influence of coordinate errors in a single image, further improving accuracy.

[0043] Optionally, the fitted coordinates of the location points determined based on the function expression can be adjusted, and the total error can be obtained by calculating the sum of the projection error and the laser geometric constraint parameters. The laser geometric constraint parameters are determined by substituting the fitted coordinates of the location points into the function expression. The fitted coordinates of the location point that minimizes the total error are then determined as the geographic coordinates of the location point. It should be noted that the laser geometric constraint parameters and the projection error can be weighted and summed to obtain the total error, and the weights can be set according to specific circumstances. The laser geometric constraint parameters can be 0 or non-zero; two specific examples are given here to illustrate this.

[0044] Optionally, in the first example, the map feature includes a preset surface with a laser geometric constraint parameter of 0, indicating that the location point lies within the preset surface. Optionally, taking a ground feature as an example, the laser geometric constraint parameter is 0, indicating that the location point lies on the ground. The preset surface can be the ground or the surface of a sign, etc. The laser geometric constraint parameter is obtained by substituting the fitted coordinates of the location point into a function expression. When the laser geometric constraint parameter is 0, it means that the fitted coordinates of the location point conform to the function expression, i.e., the location point is within the map feature. Further, nonlinear optimization is performed based on the projection error and the set laser geometric constraint parameter. The geographic coordinates of the location point of the map feature are determined according to the result of the nonlinear optimization, including: adjusting the fitted coordinates of the location point determined based on the function expression so that the fitted coordinates of the location point satisfy the function expression, and determining the projection error; the fitted coordinates of the location point that minimizes the projection error are determined as the geographic coordinates of the location point. It should be noted that if the fitted coordinates of the location point satisfy the functional expression of the map feature, the result is 0, and the laser geometric constraint parameter is 0; if the fitted coordinates of the location point do not satisfy the functional expression of the map feature, the result is not 0, and the laser geometric constraint parameter can be a positive number greater than 0 or positive infinity. In this implementation, the fitted coordinates of the location point need to satisfy the functional expression of the map feature, that is, the laser geometric constraint parameter is controlled to be 0. Of course, this is just an example.

[0045] Optionally, in the second example, the laser geometric constraint parameter can be non-zero. The sum of the errors between the fitted coordinates of the location point determined by the function expression and the initial geographic coordinates of the location point is used as the laser geometric constraint parameter. This sum of errors can be obtained by summing the errors between the fitted coordinates and the initial geographic coordinates of the location point in multiple images. The fitted coordinates of the location point determined by the function expression are adjusted, and then the projection error and the laser geometric constraint parameter are calculated. A weighted sum of the projection error and the laser geometric constraint parameter is performed to obtain the total error. The fitted coordinates of the location point that minimizes the total error are determined as the geographic coordinates of the location point.

[0046] It should also be noted that, combining steps 205 and 206, when adjusting the fitted coordinates of a position point, the fitted coordinates of that point can conform to a functional expression. Each time the fitted coordinates of a position point are adjusted, the projection error and laser geometric constraint parameters need to be calculated based on those fitted coordinates (this calculation is not required in the first example), until the fitted coordinates of the position point with the minimum total error are found.

[0047] After determining the geographic coordinates of a location point, the set of geographic coordinates of map elements can be determined based on the geographic coordinates of at least one location point, thereby creating an electronic map.

[0048] By combining the image coordinates of shape points in the camera-captured images and the fitting of position points using the function expression of the laser point cloud, the projected coordinates of the shape points are obtained through projection. Then, nonlinear optimization is performed. By combining the accuracy of the camera-captured images and the accuracy of the laser point cloud, the accuracy and update rate of the electronic map are improved while maintaining a low cost.

[0049] Based on the map feature generation method described in steps 201-206 above, three specific application scenarios are listed here to further illustrate the method.

[0050] Optionally, in the first application scenario, the map element can be the ground. An electronic device can capture at least one image of the ground (i.e., a set of images) from different angles using a camera. For each image, features of the map element are extracted through image recognition. Map elements can be lane lines, zebra crossings, turn arrows, etc., and the image coordinates of the shape points of the map element can be obtained. According to the principles of photography, a ray is drawn from the optical center, passing through the shape point on the imaging plane and intersecting the ground. The intersection point P is the location point corresponding to that shape point in actual physical space. The initial geographic coordinates of the location point are calculated as follows: Figure 3 As shown. Since the ground may not be flat, but uneven due to speed bumps, local depressions, or protrusions, the ground's functional expression (i.e., the functional expression of map elements) is not fixed. Relying solely on a pre-defined ground functional expression is insufficient for accuracy; reconstruction is necessary. Specifically, based on the initial geographic coordinates of a location point, its neighborhood is determined in the laser point cloud. The ground's functional expression is then fitted using laser points within this neighborhood. Furthermore, the fitted coordinates of the location point in the actual physical space can be determined using the fitted ground functional expression. Performing this process for each image yields the image coordinates of shape points and the corresponding fitted coordinates of location points in each image.

[0051] Nonlinear optimization is performed using the image coordinates of shape points and the fitted coordinates of position points. Specifically, nonlinear optimization can be performed using Formula 1, which is:

[0052] argmin∑reprojection(P,X i ) 2 +diff(P,Lidar)

[0053] Where P is the fitted coordinate of the location point, X i Let P, X be the image coordinates of the shape point in the i-th image. i ) 2The error between the coordinates of the point projected onto the imaging plane and the coordinates of the shape point (i.e., projection error) is represented by `argmin()`, which finds the coordinates of point P that minimize the value of the expression within the parentheses. `diff(P,Lidar)` represents the laser geometric constraint parameter; when `f(P) = 0`, `diff(P,Lidar) = 0`, otherwise `diff(P,Lidar)` is positive infinity. `f()` is the function expression for the ground; `f(P) = 0` indicates that the coordinates of point P are within the ground (i.e., the preset surface, map element), thus satisfying the ground function expression. Moving the position of P, i.e., transforming the coordinates of P, gradually pulls the position of point P to the position with the highest consistency with the image observation during the optimization iteration, thereby obtaining a more accurate reconstruction result.

[0054] Optionally, in the second application scenario, the map element can be a sign plane, such as a road sign plane or a traffic sign plane. Features of the map element are extracted through image recognition; the map element can be traffic signs, text, etc., and the image coordinates of the shape points of the map element can be obtained. In another image, the electronic device can obtain at least one image (a set of images) by taking pictures of the sign plane from different angles. For each image, because the shooting angles are different, the coordinates of the shape points are different in different images. For example, as... Figure 4 As shown, image recognition is performed on image A to determine the coordinates of shape points in image A, and image recognition is performed on image B to determine the coordinates of shape points in image B. Images A and B can be two adjacent images, or the first and last frames of a set of images. Choosing the first and last frames yields the longest photographic baseline, improving the accuracy of triangulation. Triangulation is performed on each shape point in image A and image B to determine the initial geographic coordinates of the location points. This allows for the determination of the initial geographic coordinates of multiple location points on the sign plane. The neighborhood of each location point in the laser point cloud is determined, and the sign plane is fitted using the laser points in the neighborhood of each location point. Using Formula 1, nonlinear optimization is performed, resulting in a more accurate reconstruction result (i.e., the geographic coordinates of the location points).

[0055] Optionally, in the third application scenario, the electronic device can acquire at least one image (i.e., a set of images) by taking pictures from different angles using a camera. For each image, the image coordinates of a shape point in a map feature can be determined. In this application scenario, the map feature is manually defined; for example, the map feature could be a line segment. The initial coordinates of the vertex and the initial coordinates of the base of a line segment in the image can be extracted. This can also be understood as determining two shape points in the image, using the same triangulation method as in the second application scenario, such as... Figure 4As shown, the image coordinates of the shape points are determined in the two images respectively, and then the initial geographic coordinates of the corresponding location points are determined. Since there are two shape points in this application scenario, two location points P1 and P2 can be determined. Based on the initial geographic coordinates of location points P1 and P2, the neighborhood of P1 and the neighborhood of P2 in the laser point cloud are determined. P1P2 is fitted using the laser points in the neighborhood, and the fitted line segment is denoted as P1`P2`. Combining the image coordinates of the shape points and the reconstructed line segment, Formula 2 is constructed for nonlinear optimization, as follows:

[0056]

[0057] Among them, X1 i X2 i This represents the line segment formed by two shape points in the i-th frame of the image, such as... Figure 5 As shown, X1X2 in each frame of the image is projected onto the same P1P2 in the actual physical space. `reprojection()` represents the error between the position point projected onto the imaging plane and the shape point, i.e., the line segment of P1P2 projected onto the imaging plane and X1. i X2 i The error (i.e. projection error) is defined as follows: P1`P2` represents the line segment (i.e. the fitted coordinates of the location point) obtained by fitting the laser point cloud; P1P2 represents the line segment (i.e. the initial geographic coordinates of the location point) determined by the image; diff() represents the error between P1P2 and P1`P2` (i.e. the laser geometric constraint parameters); and w1 and w2 are two weights.

[0058] The map feature generation method provided in this application identifies an image to obtain the image coordinates of shape points of map features recorded in the image; based on the image coordinates of the shape points, performs visual 3D reconstruction to obtain the initial geographic coordinates of the corresponding location points in the actual physical space; based on the initial geographic coordinates of the location points, determines the neighborhood of the location points in the laser point cloud corresponding to the image; based on the laser points in the neighborhood, fits a function expression representing the shape of the map feature; based on the function expression and the image coordinates of the shape points, determines the projection error of the shape points; based on the projection error and set laser geometric constraint parameters, performs nonlinear optimization, and determines the geographic coordinates of the location points of the map features based on the result of the nonlinear optimization. By combining the image coordinates of the shape points in the image and the projection error determined by projecting using the function expression fitted by the laser point cloud, nonlinear optimization is performed. By combining the accuracy of the image captured by the camera and the accuracy of the laser point cloud, the accuracy and update rate of the electronic map are improved while maintaining a low cost.

[0059] Example 2

[0060] Based on the method described in Embodiment 1 above, Embodiment 2 of this application provides a map feature generation apparatus for executing the method described in Embodiment 1 above, with reference to... Figure 6 As shown, the map feature generation device 60 includes:

[0061] The recognition module 601 is used to recognize the image to obtain the image coordinates of the shape points of the map features recorded in the image;

[0062] The reconstruction module 602 is used to perform visual 3D reconstruction based on the image coordinates of the shape points to obtain the initial geographic coordinates of the corresponding location points of the shape points in the actual physical space.

[0063] The laser point cloud module 603 is used to determine the neighborhood of the location point in the laser point cloud corresponding to the image based on the initial geographic coordinates of the location point;

[0064] Function module 604 is used to fit a function expression representing the shape of map features based on laser points in the neighborhood;

[0065] Error module 605 is used to determine the projection error of shape points based on function expressions and image coordinates of shape points;

[0066] The nonlinear optimization module 606 is used to perform nonlinear optimization based on projection error and set laser geometric constraint parameters, and to determine the geographic coordinates of the location points of map elements based on the results of nonlinear optimization.

[0067] Optionally, in a specific example, the map feature is a ground feature, and the reconstruction module 602 is used to calculate the coordinates of the intersection point of the ray passing through the shape point from the optical center and the preset surface based on the image coordinates of the shape point and the initial function expression of the preset surface, as the initial geographic coordinates of the corresponding location point of the shape point in the actual physical space.

[0068] Optionally, in one specific example, the preset surface is the ground, and the laser geometric constraint parameter is 0, used to indicate that the location point falls on the ground.

[0069] Optionally, in a specific example, the map element is a sign element or a pole element. The reconstruction module 602 is used to determine the coordinates of the intersection of the first ray and the second ray as the initial geographic coordinates of the shape point in the actual physical space, based on the image coordinates of the shape point in the first image, using a ray extending outward from the first optical center through the shape point in the first image as the first ray, and the image coordinates of the shape point in the second image, using a ray extending outward from the second optical center through the shape point in the second image as the second ray. The first image is the image of the sign element observed in the first frame, and the second image is the image of the sign element or pole element observed in the last frame.

[0070] Optionally, in a specific example, if the map feature is a pole feature, then the shape points include: shape points corresponding to the vertices and shape points corresponding to the bases of the poles.

[0071] Optionally, in a specific example, the error module 605 is used to calculate the coordinates of the intersection point of the straight line passing through the position point and the optical center and the imaging plane based on the fitted coordinates of the position point determined by the function expression, as the projection coordinates of the shape point; and to calculate the sum of the errors of the image coordinates and projection coordinates of the shape point in at least one image as the projection error.

[0072] Optionally, in a specific example, the nonlinear optimization module 606 is used to adjust the fitted coordinates of the location point determined based on the function expression, so that the fitted coordinates of the location point satisfy the function expression, and to determine the projection error; the fitted coordinates of the location point that satisfies the minimum projection error are determined as the geographic coordinates of the location point.

[0073] Optionally, in a specific example, the nonlinear optimization module 606 is used to adjust the fitted coordinates of the location point determined based on the function expression, calculate the sum of the projection error and the laser geometric constraint parameters to obtain the total error, and determine the laser geometric constraint parameters based on the result obtained by substituting the fitted coordinates of the location point into the function expression; and determine the fitted coordinates of the location point that satisfies the minimum total error as the geographic coordinates of the location point.

[0074] Optionally, in one specific example, the error module 605 is further configured to use the sum of the errors between the fitted coordinates of the location point determined based on the function expression and the initial geographic coordinates of the location point as laser geometric constraint parameters.

[0075] The map element generation apparatus provided in this application identifies an image to obtain the image coordinates of shape points of map elements recorded in the image; based on the image coordinates of the shape points, performs visual 3D reconstruction to obtain the initial geographic coordinates of the corresponding location points in the actual physical space; based on the initial geographic coordinates of the location points, determines the neighborhood of the location points in the laser point cloud corresponding to the image; based on the laser points in the neighborhood, fits a function expression representing the shape of the map element; based on the function expression and the image coordinates of the shape points, determines the projection error of the shape points; based on the projection error and set laser geometric constraint parameters, performs nonlinear optimization, and determines the geographic coordinates of the location points of the map elements based on the result of the nonlinear optimization. By combining the image coordinates of the shape points in the image and the projection error determined by projecting using the function expression fitted by the laser point cloud, nonlinear optimization is performed. By combining the accuracy of the image captured by the camera and the accuracy of the laser point cloud, the accuracy and update rate of the electronic map are improved while maintaining a low cost.

[0076] Example 3

[0077] Based on the method described in Embodiment 1 above, Embodiment 3 of this application provides an electronic device for executing the method described in Embodiment 1 above, with reference to... Figure 7 The diagram shows a structural schematic of an electronic device according to Embodiment 3 of this application. The specific embodiments of this application do not limit the specific implementation of the electronic device.

[0078] like Figure 7 As shown, the electronic device 70 may include: a processor 702, a communications interface 704, a memory 706, and a communications bus 708.

[0079] in:

[0080] The processor 702, communication interface 704, and memory 706 communicate with each other via communication bus 708.

[0081] Communication interface 704 is used to communicate with other electronic devices or servers.

[0082] The processor 702 is used to execute program 710, specifically to perform the relevant steps in the above-described map feature generation method embodiment.

[0083] Specifically, program 710 may include program code that includes computer operation instructions.

[0084] The processor 702 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application. The smart device includes one or more processors, which may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0085] Memory 706 is used to store program 710. Memory 706 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0086] Specifically, program 710 can be used to cause processor 702 to execute the map feature generation method described in Embodiment 1. The specific implementation of each step in program 710 can be found in the corresponding descriptions of the steps and units in the above-described map feature generation method embodiments, and will not be repeated here. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding process descriptions in the foregoing method embodiments, and will not be repeated here.

[0087] The electronic device provided in this application identifies an image to obtain the image coordinates of shape points of map elements recorded in the image; based on the image coordinates of the shape points, performs visual 3D reconstruction to obtain the initial geographic coordinates of the corresponding location points in the actual physical space; based on the initial geographic coordinates of the location points, determines the neighborhood of the location points in the laser point cloud corresponding to the image; based on the laser points in the neighborhood, fits a function expression representing the shape of the map elements; based on the function expression and the image coordinates of the shape points, determines the projection error of the shape points; based on the projection error and set laser geometric constraint parameters, performs nonlinear optimization, and determines the geographic coordinates of the location points of the map elements based on the result of the nonlinear optimization. By combining the image coordinates of the shape points in the image and the projection error determined by projecting using the function expression fitted by the laser point cloud, nonlinear optimization is performed. By combining the accuracy of the image captured by the camera and the accuracy of the laser point cloud, the accuracy and update rate of the electronic map are improved while maintaining a low cost.

[0088] Example 4

[0089] Based on the method described in Embodiment 1 above, Embodiment 4 of this application provides a computer storage medium storing a computer program thereon, which, when executed by a processor, implements the method described in Embodiment 1.

[0090] Example 5

[0091] Based on the method described in Embodiment 1 above, Embodiment 4 of this application provides a computer program product that, when executed by a processor, implements the method described in Embodiment 1.

[0092] It should be noted that, depending on the implementation needs, the various components / steps described in the embodiments of this application can be broken down into more components / steps, or two or more components / steps or parts of the operation of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of this application.

[0093] The methods described in the embodiments of this application can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code downloaded over a network that is originally stored in a remote recording medium or a non-transitory machine-readable medium and will be stored in a local recording medium. Thus, the methods described herein can be processed by software stored on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an ASIC or FPGA). It is understood that the computer, processor, microprocessor controller, or programmable hardware includes storage components (e.g., RAM, ROM, flash memory, etc.) capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the navigation methods described herein are implemented. Furthermore, when a general-purpose computer accesses the code used to implement the navigation methods shown herein, the execution of the code transforms the general-purpose computer into a dedicated computer for executing the navigation methods shown herein.

[0094] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this application.

[0095] The above embodiments are only used to illustrate the embodiments of this application, and are not intended to limit the embodiments of this application. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of this application. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of this application, and the patent protection scope of the embodiments of this application should be defined by the claims.

Claims

1. A method for generating map features, wherein, include: Identify the image to obtain the image coordinates of the shape points of the map features recorded in the image; Based on the image coordinates of the shape point, a visual 3D reconstruction is performed to obtain the initial geographic coordinates of the corresponding location point of the shape point in the actual physical space. Based on the initial geographic coordinates of the location point, determine the neighborhood of the location point in the laser point cloud corresponding to the image; Based on the laser points in the neighborhood, fit a functional expression representing the shape of the map feature; Based on the function expression and the image coordinates of the shape point, determine the projection error of the shape point; Based on the projection error and the set laser geometric constraint parameters, nonlinear optimization is performed, and the geographic coordinates of the location points of the map elements are determined according to the results of the nonlinear optimization.

2. The method according to claim 1, wherein, The map elements are ground features. The process of performing visual 3D reconstruction based on the image coordinates of the shape points to obtain the initial geographic coordinates of the corresponding locations of the shape points in actual physical space includes: Based on the image coordinates of the shape point and the initial function expression of the preset surface, the coordinates of the intersection point of the ray passing through the optical center from the shape point and the preset surface are calculated, which are used as the initial geographic coordinates of the shape point in the actual physical space.

3. The method according to claim 2, wherein, The preset surface is the ground, and the laser geometric constraint parameter is 0, which is used to indicate that the position point falls on the ground.

4. The method according to claim 1, wherein, The map elements are sign elements or pole elements. The process of performing visual 3D reconstruction based on the image coordinates of the shape points to obtain the initial geographic coordinates of the corresponding locations of the shape points in actual physical space includes: Based on the image coordinates of the shape point in the first image, the ray extending outward from the shape point in the first image through the first optical center is taken as the first ray. Based on the image coordinates of the shape point in the second image, the ray extending outward from the shape point in the second image through the second optical center is taken as the second ray. The coordinates of the intersection of the first ray and the second ray are determined as the initial geographic coordinates of the shape point in the actual physical space. The first image is the image of the sign element observed in the first frame, and the second image is the image of the sign element or pole element observed in the last frame.

5. The method according to claim 4, wherein, If the map element is a pole element, then the shape points include: the shape points corresponding to the vertices and the shape points corresponding to the bases of the pole.

6. The method according to claim 1, wherein, The determination of the projection error of the shape point based on the function expression and the image coordinates of the shape point includes: Based on the fitted coordinates of the position point determined using the function expression, calculate the coordinates of the intersection point of the straight line passing through the position point and the optical center with the imaging plane, and use them as the projected coordinates of the shape point; The projection error is calculated as the sum of the errors between the image coordinates and the projected coordinates of a shape point in at least one of the images.

7. The method according to claim 6, wherein, The nonlinear optimization based on the projection error and the set laser geometric constraint parameters, and the determination of the geographic coordinates of the location points of the map features based on the results of the nonlinear optimization, includes: Adjust the fitted coordinates of the position points determined based on the function expression so that the fitted coordinates of the position points satisfy the function expression, and determine the projection error; The fitted coordinates of the location point that minimizes the projection error are determined as the geographic coordinates of the location point.

8. The method according to any one of claims 1-5, wherein, The nonlinear optimization based on the projection error and the set laser geometric constraint parameters, and the determination of the geographic coordinates of the location points of the map features based on the results of the nonlinear optimization, includes: Adjust the fitted coordinates of the position point determined based on the function expression, calculate the sum of the projection error and the laser geometric constraint parameters to obtain the total error, and determine the laser geometric constraint parameters based on the result obtained by substituting the fitted coordinates of the position point into the function expression; The fitted coordinates of the location point that minimizes the total error are determined as the geographic coordinates of the location point.

9. The method according to claim 8, wherein, The method further includes: The sum of the errors between the fitted coordinates of the location point determined based on the function expression and the initial geographic coordinates of the location point is used as the laser geometric constraint parameter.

10. A computer program product, which, when executed by a processor, implements the map feature generation method as described in any one of claims 1-8.

Images