Map construction method and device

Abstract

The embodiment of the invention provides a map construction method and device, and relates to the technical field of data processing.The method comprises the steps that sensor data collected by sensors carried on a carrier in the driving process of the carrier are obtained, the sensors carried on the carrier comprise a visual sensor and a pose sensor, and the visual sensor and the pose sensor are connected with the visual sensor; the pose sensor is a non-visual sensor used for performing synchronous localization and map reconstruction (SLAM); sLAM is carried out based on the obtained sensor data to obtain reconstruction information, and map element detection is carried out on an image collected by the visual sensor in the obtained sensor data; constructing road surface description information of a road on which the carrier passes based on the reconstruction information; based on the road surface description information, determining element positions of the detected map elements in the road; and constructing a map for the road based on the determined element positions. By applying the scheme provided by the embodiment of the invention, the construction efficiency of the map can be improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a map construction method and device. Background Art

[0002] Vehicle autonomous driving and vehicle navigation are inseparable from maps, so maps need to be built. The map of an area reflects the three-dimensional environmental information of the area.

[0003] When constructing a map in the related technology, a vehicle equipped with a lidar travels on the road, so that the lidar can collect point cloud data of the area through which the vehicle has traveled. The staff then annotates the point cloud data with road objects and lane-level topological relationships, and the map construction equipment constructs the map based on the annotation results.

[0004] Although the above method can successfully construct a map, manual labeling is inefficient, resulting in low efficiency in map construction. Summary of the invention

[0005] The purpose of the embodiment of the present application is to provide a map construction method and device to improve the efficiency of map construction. The specific technical solution is as follows:

[0006] In a first aspect, an embodiment of the present application provides a map construction method, the method comprising:

[0007] Obtaining sensor data collected by sensors carried on the vehicle during the vehicle's travel, wherein the sensors carried on the vehicle include: a visual sensor and a posture sensor, wherein the posture sensor is: a non-visual sensor for performing simultaneous positioning and map reconstruction (SLAM);

[0008] Performing SLAM based on the obtained sensor data to obtain reconstruction information, and performing map element detection on the image collected by the visual sensor in the obtained sensor data;

[0009] Constructing road surface description information of the road traveled by the vehicle based on the reconstruction information;

[0010] Based on the road surface description information, determining an element position of the detected map element in the road;

[0011] Based on the determined element positions, a map for the road is constructed.

[0012] In one embodiment of the present application, the reconstruction information includes: road feature points, and the road description information of the road traveled by the vehicle based on the reconstruction information includes:

[0013] Determining road surface parameters of the road traveled by the vehicle based on at least three non-co-route road surface feature points;

[0014] According to the determined road surface parameters, a road surface equation of the road is generated as road surface description information of the road.

[0015] In one embodiment of the present application, the reconstruction information includes: the vehicle posture of the vehicle, and the road surface description information of the road traveled by the vehicle based on the reconstruction information includes:

[0016] Based on the vehicle posture, determining a road surface normal vector of the road traveled by the vehicle;

[0017] Determine a real road surface point on the road where the vehicle travels based on the vehicle posture and a pre-acquired height of the vehicle relative to the road surface;

[0018] Determining a road surface intercept parameter of the road according to the road surface normal vector and a real road surface point;

[0019] Based on the road surface intercept parameter and the normal vector, a road surface equation of the road is generated as road surface description information of the road.

[0020] In one embodiment of the present application, determining the element position of the detected map element in the road based on the road surface description information includes:

[0021] Based on the sensor posture when the visual sensor collects the image and the sensor internal parameters of the visual sensor, a conversion relationship between the pixel position of the pixel point in the image collected by the visual sensor and the spatial position in the real three-dimensional space is obtained;

[0022] Based on the obtained conversion relationship, determining the projection ray of the pixel point describing the map element in the image collected by the visual sensor in the real three-dimensional space;

[0023] Determine an intersection point between the projection ray and the road surface described by the road surface description information;

[0024] Based on the determined intersection point, an element position of a map element in the road in the image collected by the visual sensor is determined.

[0025] In one embodiment of the present application, obtaining the conversion relationship between the pixel position of a pixel point in the image collected by the visual sensor and the spatial position in the real three-dimensional space based on the sensor posture when the visual sensor collects the image and the sensor internal parameter of the visual sensor includes:

[0026] Based on the sensor posture when the visual sensor collects images and the sensor internal parameters of the visual sensor, the conversion relationship between the pixel position of the pixel point in the image collected by the visual sensor and the spatial position in the real three-dimensional space is obtained by back projection.

[0027] In one embodiment of the present application, the posture sensor includes: an inertial sensor and a positioning sensor;

[0028] The step of performing SLAM based on the obtained sensor data to obtain reconstruction information includes:

[0029] SLAM is performed based on the image collected by the visual sensor, the inertial information collected by the inertial sensor, and the positioning information collected by the positioning sensor in the sensor data to obtain reconstruction information.

[0030] In one embodiment of the present application, the SLAM is performed based on the image collected by the visual sensor, the inertial information collected by the inertial sensor, and the positioning information collected by the positioning sensor in the sensor data to obtain reconstruction information, including:

[0031] Extracting visual feature points of the image collected by the visual sensor in the obtained sensor data, and matching the visual feature points of adjacent images;

[0032] Based on the matching result, determining a first relative posture when the visual sensor acquires the adjacent images and a spatial position of the matched visual feature point in the coordinate system of the visual sensor;

[0033] Determine, based on the inertial information collected by the inertial sensor in the obtained sensor data, a second relative posture when the visual sensor collects the adjacent images;

[0034] Determine the global position of the visual sensor when acquiring images and the driving speed of the vehicle based on the positioning information collected by the positioning sensor in the obtained sensor data;

[0035] SLAM is performed according to the first relative posture, the determined spatial position, the second relative posture, the global posture and the driving speed to obtain reconstruction information.

[0036] In one embodiment of the present application, constructing a map for the road based on the determined element positions includes:

[0037] Performing position fusion on element positions corresponding to the same map element in the image collected by the visual sensor to obtain a fused position of the detected map element;

[0038] Based on the obtained fused positions, a map for the road is constructed.

[0039] In one embodiment of the present application, the visual sensor includes: a monocular camera.

[0040] In a second aspect, an embodiment of the present application provides a map construction device, the device comprising:

[0041] A sensor data acquisition module, used to obtain sensor data collected by sensors carried on the vehicle during the vehicle's driving process, wherein the sensors carried on the vehicle include: visual sensors and posture sensors, wherein the posture sensors are: non-visual sensors used for simultaneous positioning and map reconstruction SLAM;

[0042] A reconstruction information acquisition module is used to perform SLAM based on the obtained sensor data to obtain reconstruction information, and to perform map element detection on the image collected by the visual sensor in the obtained sensor data;

[0043] A road surface description information construction module, used to construct road surface description information of the road traveled by the vehicle based on the reconstruction information;

[0044] An element position determination module, used to determine the element position of the detected map element in the road based on the road description information;

[0045] A map building module is used to build a map for the road based on the determined element positions.

[0046] In one embodiment of the present application, the reconstruction information includes: road surface feature points, and the road surface description information construction module is specifically used to determine the road surface parameters of the road traveled by the vehicle based on at least three non-co-linear road surface feature points; based on the determined road surface parameters, generate the road surface equation of the road as the road surface description information of the road.

[0047] In one embodiment of the present application, the reconstruction information includes: the vehicle posture of the vehicle, the road description information construction module, which is specifically used to determine the road surface normal vector of the road traveled by the vehicle based on the vehicle posture; determine the real road surface point in the road traveled by the vehicle based on the vehicle posture and the pre-obtained height of the vehicle relative to the road surface; determine the road surface intercept parameter of the road based on the road surface normal vector and the real road surface point; and generate the road surface equation of the road based on the road surface intercept parameter and the normal vector as the road surface description information of the road.

[0048] In one embodiment of the present application, the element position determination module includes:

[0049] A conversion relationship acquisition submodule, used to obtain a conversion relationship between a pixel position of a pixel point in an image acquired by the visual sensor and a spatial position in a real three-dimensional space based on a sensor posture when the visual sensor acquires an image and a sensor internal parameter of the visual sensor;

[0050] A projection ray determination submodule, used to determine the projection ray of the pixel points describing the map elements in the image collected by the visual sensor in the real three-dimensional space based on the obtained conversion relationship;

[0051] An intersection determination submodule, used to determine the intersection of the projection ray and the road surface described by the road surface description information;

[0052] The element position determination submodule is used to determine the element position of the map element in the image collected by the visual sensor on the road based on the determined intersection point.

[0053] In one embodiment of the present application, the conversion relationship acquisition submodule is specifically used to obtain the conversion relationship between the pixel position of a pixel point in the image captured by the visual sensor and the spatial position in the real three-dimensional space by reverse projection based on the sensor posture when the visual sensor captures the image and the sensor intrinsic parameters of the visual sensor.

[0054] In one embodiment of the present application, the posture sensor includes: an inertial sensor and a positioning sensor; the reconstruction information acquisition module includes:

[0055] The reconstruction information acquisition submodule is used to perform SLAM based on the image collected by the visual sensor, the inertial information collected by the inertial sensor, and the positioning information collected by the positioning sensor in the sensor data to obtain reconstruction information.

[0056] In one embodiment of the present application, the reconstruction information acquisition submodule is specifically used to extract visual feature points of the image captured by the visual sensor in the obtained sensor data, and match the visual feature points of adjacent images; based on the matching results, determine the first relative posture of the visual sensor when collecting the adjacent images and the spatial position of the matched visual feature points in the visual sensor coordinate system; based on the inertial information collected by the inertial sensor in the obtained sensor data, determine the second relative posture of the visual sensor when collecting the adjacent images; based on the positioning information collected by the positioning sensor in the obtained sensor data, determine the global posture of the visual sensor when collecting the image and the driving speed of the vehicle; perform SLAM according to the first relative posture, the determined spatial position, the second relative posture, the global posture and the driving speed to obtain reconstruction information.

[0057] In one embodiment of the present application, the map construction module is specifically used to perform position fusion on element positions corresponding to the same map element in the image captured by the visual sensor to obtain the fused position of the detected map element; and construct a map for the road based on the obtained fused position.

[0058] In one embodiment of the present application, the visual sensor includes: a monocular camera.

[0059] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0060] Memory, used to store computer programs;

[0061] The processor is used to implement the method described in the first aspect when executing the program stored in the memory.

[0062] In a fourth aspect, an embodiment of the present application provides a vehicle, wherein the vehicle is equipped with the electronic device described in the third aspect.

[0063] In a fifth aspect, an embodiment of the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the method described in the first aspect.

[0064] As can be seen from the above, when the scheme provided by the embodiment of the present application is used to construct a map, the sensor data collected by the sensors carried by the vehicle during the driving process of the vehicle is obtained, and then SLAM can be performed based on the obtained sensor data to obtain reconstruction information, and the road surface description information of the road traveled by the vehicle can be constructed based on the reconstruction information, so that the element position of the map element in the image collected by the visual sensor in the road can be determined based on the road surface description information, and then the map for the road can be successfully constructed based on the determined element position. It can be seen that the above-mentioned map construction process can be realized based on the sensor data collected by the sensors carried by the vehicle, without the need for manual annotation by the staff, which reduces the workload of the map construction process and improves the efficiency of map construction.

[0065] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0067] Figure 1A schematic diagram of the process of the first map construction method provided in the embodiment of the present application;

[0068] Figure 2 A schematic diagram of a normal vector determination process provided in an embodiment of the present application;

[0069] Figure 3 A schematic diagram of a second map construction method provided in an embodiment of the present application;

[0070] Figure 4 A schematic diagram of an intersection provided in an embodiment of the present application;

[0071] Figure 5 A schematic diagram of the structure of a map construction device provided in an embodiment of the present application;

[0072] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0073] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0074] First, the execution subject of the solution provided in the embodiment of the present application is described.

[0075] The execution subject of the solution provided in the embodiment of the present application is: any electronic device with data processing, storage and other functions. For example, it can be a carrier with data processing, storage and other functions, or it can be a background server.

[0076] The map construction solution provided in the embodiment of the present application is described in detail below.

[0077] See also Figure 1 , is a flow chart of a first map construction method provided in an embodiment of the present application, the method comprising the following steps S101 to S106.

[0078] Step S101: obtaining sensor data collected by sensors mounted on the vehicle during the vehicle's driving process.

[0079] The above-mentioned vehicle may be a vehicle, an intelligent robot, etc., which is not limited in the embodiments of the present application.

[0080] Among them, the sensors carried by the vehicle include: visual sensors and posture sensors.

[0081] The above-mentioned visual sensor may include a monocular camera, a three-dimensional depth sensor, etc. Since the cost of a monocular camera is relatively low, the use of a monocular camera can save the cost of a map construction solution.

[0082] The above-mentioned posture sensor may include any non-visual sensor used for simultaneous localization and mapping SLAM (Simultaneous Localization And Mapping), which is not limited in the embodiments of the present application.

[0083] For example, the above-mentioned posture sensor may include a radar sensor, an inertial sensor, a positioning sensor, etc. Among them, the above-mentioned inertial sensor may be an IMU (Inertial Measurement Unit), an acceleration sensor, a gyroscope, etc., and the above-mentioned positioning sensor may be a GPS (Global Positioning System) sensor, a Beidou satellite navigation system sensor, etc.

[0084] Depending on the type of sensor, the sensor data collected by the sensor is also different.

[0085] For example, the sensor data collected by the visual sensor can be an image, the sensor data collected by the inertial sensor can be posture data such as acceleration and angular velocity, and the sensor data collected by the positioning sensor can be positioning data such as latitude and longitude, height, and speed.

[0086] The timing of obtaining sensor data in this step is described below.

[0087] In one case, the sensor data collected by the above sensors during the driving of the vehicle can be obtained in real time. In this case, as the sensor continuously collects data, the sensor data collected by the sensor can be continuously obtained and cached.

[0088] In another case, all sensor data collected by the above sensors during the vehicle traveling a preset distance can be obtained. The above preset distance can be set by the staff based on experience, for example, it can be 20 meters, 50 meters, etc.

[0089] Step S102: Perform SLAM based on the acquired sensor data to obtain reconstruction information.

[0090] The timing for executing this step is described below.

[0091] In one case, if the obtained sensor data is all sensor data collected by the sensor during the vehicle traveling a preset distance, this step can be directly executed.

[0092] In another case, if the obtained sensor data is sensor data acquired in real time, this step can be performed based on the cached sensor data.

[0093] For example, the sensor data of the latest cached preset data volume can be determined from the cached data. In this way, the determined sensor data is the sensor data collected during the driving process of the vehicle in the previous period of time before the current moment, and this step can be performed based on the determined sensor data.

[0094] Specifically, the method of obtaining the reconstruction information in this step is different depending on the type of sensor, which is not limited in the embodiment of the present application and is introduced below with examples.

[0095] In one embodiment, when the sensors carried on the vehicle include a monocular camera and a lidar, SLAM can be performed based on the image captured by the monocular camera and the distance information collected by the lidar in the sensor data to obtain road surface feature points in the road as reconstruction information; the vehicle's posture can also be determined based on the obtained road surface feature points as reconstruction information.

[0096] In another implementation, when the posture sensor includes an inertial sensor and a positioning sensor, SLAM can be performed based on the image collected by the visual sensor, the inertial information collected by the inertial sensor, and the positioning information collected by the positioning sensor in the sensor data to obtain reconstruction information. The specific implementation method is detailed in the subsequent embodiments and will not be described in detail here.

[0097] In this way, on the basis of the images collected by the visual sensor, the inertial information collected by the inertial sensor and the positioning information collected by the positioning sensor are combined for SLAM. It can be seen that this not only takes into account the visual information, but also the inertial information and positioning information, so that more accurate reconstruction information can be obtained based on a variety of different types of information.

[0098] Step S103: performing map element detection on the image collected by the visual sensor in the obtained sensor data.

[0099] The above map elements can be any type of objects in the image.

[0100] For example, it may be a lane line, a stop line, an arrow, a zebra crossing, a roadside tree, an obstacle on the road, etc., which is not limited in the embodiments of the present application.

[0101] Specifically, the following method may be used to perform map element detection on images collected by a visual sensor.

[0102] In one implementation, a deep learning algorithm may be used to detect map elements in the above image.

[0103] For example, the above-mentioned deep learning algorithm can be an algorithm for object detection based on the Fast R-CNN model, an algorithm for object detection based on the YoLo model, an SSD (Single Shot MultiBox Detector) algorithm, etc.

[0104] In another implementation, the above image may be input into a pre-trained map element detection model to obtain the map elements in the image output by the above model.

[0105] The map element detection model may be a model for detecting map elements in an image obtained by training a preset neural network model using a sample image as input and the annotation boxes of the map elements in the sample image as training annotations.

[0106] Step S104: Constructing road surface description information of the road traveled by the vehicle based on the reconstruction information.

[0107] Based on the different reconstruction information, the methods of constructing the road surface description information in this step are also different, which are explained below.

[0108] In one embodiment, when the reconstruction information includes road surface feature points, the road surface parameters of the road traveled by the vehicle can be determined based on at least three non-co-linear road surface feature points, and the road surface equation of the road can be generated according to the determined road surface parameters as the road surface description information.

[0109] Specifically, for N road feature points P = {{x i ,y i , z i For}|i∈[1,N],N>3}, based on the coplanar constraints of the feature points, the following overdetermined linear equations can be obtained:

[0110]

[0111] Among them, the above A, B, C, and D are the road surface parameters to be solved.

[0112] In this way, after solving the above-mentioned over-determined parameter equation group to obtain the road surface parameters A, B, C, and D, the plane Ax+By+Cz+D=0 is the plane equation of the plane where each feature point is located, that is, the road surface equation of the road the vehicle has traveled on. The road surface equation can also be called the road surface description information of the road the vehicle has traveled on.

[0113] By selecting at least three non-collinear road feature points, the plane equation of the plane where the feature points are located can be obtained more accurately based on the coplanar constraints of the feature points. This plane equation is also the road surface equation of the road on which the vehicle travels. Therefore, this improves the accuracy of the determined road surface equation.

[0114] In addition, based on the road feature points, a parameterized estimate of the road surface on which the vehicle travels is performed, thereby obtaining a parameterized road surface equation, which is conducive to more accurate determination of the position of map elements in the road based on the road surface equation, thereby improving the accuracy of map construction.

[0115] In another implementation, when the reconstruction information includes the vehicle posture of the vehicle, the following steps A to D may be used to obtain the above-mentioned road surface description information.

[0116] Step A: Based on the vehicle's posture, determine the road surface normal vector of the road the vehicle travels on.

[0117] Since the vehicle is traveling on the road, if a three-dimensional coordinate system is established with the vehicle's traveling direction as the X-axis direction and the vehicle's center as the origin, the Z-axis direction is perpendicular to the road surface, and the normal vector of the Z-axis is also the normal vector of the road surface. Therefore, the angle between the Z-axis and the horizontal plane can be determined, and the normal vector of the Z-axis can be determined based on the determined angle, and the normal vector is used as the normal vector of the road surface.

[0118] The following is based on Figure 2 The above process is explained more intuitively.

[0119] See also Figure 2 , which is a schematic diagram of a normal vector determination process provided in an embodiment of the present application.

[0120] Figure 2 The middle rectangular box represents the vehicle. Taking the vehicle driving on an inclined road as an example, if a three-dimensional coordinate system is established with the driving direction of the vehicle as the X-axis direction and the center of the vehicle as the origin, the direction of the Z-axis is as shown in the figure. It can be seen that the direction of the Z-axis is perpendicular to the road surface.

[0121] It should be noted that Figure 2 In the three-dimensional coordinate system shown in the figure, the direction of the y-axis is perpendicular to the paper surface, so it is not Figure 2 Shown in.

[0122] On this basis, the pitch angle α of the vehicle can be determined based on the vehicle posture, that is, Figure 2 The inclination angle of the medium inclined road surface compared to the horizontal plane is α, so the angle β of the Z axis with respect to the horizontal plane is 90°-α. Then, based on the conversion relationship between the angle of the Z axis with respect to the horizontal plane and the vector, the normal vector of the Z axis can be determined. This normal vector is the road surface normal vector of the vehicle traveling on the road.

[0123] Step B: Based on the vehicle posture and the pre-acquired height of the vehicle relative to the road surface, determine the actual road surface point on the road that the vehicle travels through.

[0124] The vehicle pose may include the position of the vehicle, and thus, based on the difference between the height component of the position and the height of the vehicle relative to the road surface, the actual road surface point on the road that the vehicle travels over may be determined.

[0125] Specifically, the difference between the height component and the above height can be calculated to obtain a new height component, and the new height component is used to replace the original height component in the above position. The obtained new position is the position of the actual road surface point on the road where the vehicle travels.

[0126] Step C: Determine the road surface intercept parameter of the road according to the road surface normal vector and the real road surface point.

[0127] Specifically, the road surface normal vector can be (A, B, C) and the coordinates of the above road surface points (x t ,y t , z t )Substitute the general plane equation Ax+By+Cz+D=0 and solve for D, which can also be called the pavement intercept parameter of the above road.

[0128] Step D: Generate a road surface equation of the road based on the road surface intercept parameter and the normal vector as the road surface description information of the road.

[0129] By substituting the road surface intercept parameter and the normal vector into the road surface equation without determining the road surface parameters, the road surface equation of the road can be obtained. The road surface equation can also be called road surface description information.

[0130] It can be seen that the road surface normal vector of the road the vehicle has traveled on can be directly determined based on the vehicle posture, and then the road surface intercept parameters of the road the vehicle has traveled on can be conveniently solved by combining the above road surface normal vector and the real road surface points. Then, the road surface equation of the road the vehicle has traveled on can be conveniently obtained based on the road surface normal vector and the road surface intercept parameters, thereby improving the efficiency of determining the road surface equation.

[0131] Moreover, in the process of determining the above-mentioned road surface equation, there is no need to use the road surface feature points obtained based on SLAM. Since the position accuracy and density of the road surface feature points are high when determining the road surface equation based on the road surface feature points, when the vehicle is moving at high speed or in a high occlusion environment, the road surface feature points are often difficult to meet the above-mentioned position accuracy and density requirements. At this time, when the road surface equation is determined based on the vehicle posture, the dependence on road surface feature points when determining the road surface equation is avoided, and the accuracy of the determined road surface equation can be further improved in an environment where the vehicle is moving at high speed or in a high occlusion environment.

[0132] In addition, based on the vehicle posture, the road surface on which the vehicle travels is parameterized and estimated, thereby obtaining a parameterized road surface equation, which is conducive to more accurate determination of the position of map elements in the road based on the road surface equation, thereby improving the accuracy of map construction.

[0133] In another embodiment, when the road surface equation of the road surface determined based on the reconstruction information represents that the road surface is a horizontal plane, the height of the vehicle relative to the road surface can be directly determined as the road surface description information.

[0134] Step S105: Determine the element position of the detected map element on the road based on the road surface description information.

[0135] Specifically, the element position of the detected map element on the road may be determined based on the following method.

[0136] In one implementation, the conversion relationship between the pixel position of a pixel point in an image and the spatial position in the real three-dimensional space can be obtained first, so that the projection ray of the pixel point describing the map element in the image in the real three-dimensional space can be determined based on the obtained conversion relationship, and then the element position of the above-mentioned map element in the road can be determined based on the intersection of the projection ray and the road surface described by the road surface description information.

[0137] For details on the implementation method, please refer to the following Figure 3 Steps S305 to S308 in the illustrated embodiment are not described in detail here.

[0138] In another embodiment, when the road description information is the height of the vehicle relative to the road surface, the position of the pixel point describing the map element in the image in the real three-dimensional space can be determined based on the conversion relationship between the pixel position of the pixel point in the image and the spatial position in the real three-dimensional space, and the element position of the detected map element in the road is determined based on the obtained position and the above-mentioned height.

[0139] Specifically, it can be seen from the above steps that when the road surface description information is the height of the vehicle relative to the road surface, the road surface representing the road is a horizontal plane, that is, the height components of the positions of points on the road surface in space are the same, and because the map elements are located on the road surface, the height components of the pixel points describing the map elements in the image are the same and related to the above height. In this way, after obtaining the position of the pixel points of the map elements in the image in the real three-dimensional space, the height component of the obtained position can be corrected according to the above height, and the corrected position is determined as the element position of the detected map element in the road.

[0140] Step S106: constructing a map for the road based on the determined element positions.

[0141] Since the obtained element position is the position of the detected map element in the road, a map for the road can be constructed based on the above element position.

[0142] Specifically, the detected map elements and the positions of the map elements may be recorded in the map, thereby obtaining a map for the road.

[0143] In one case, the element positions corresponding to the same map element in the image collected by the visual sensor can be fused to obtain the fused positions of the detected map elements, and a map for the road can be constructed based on the obtained fused positions.

[0144] Different images collected by the visual sensor may contain the same map element. In this case, the element positions corresponding to the same map element in the above images may be fused to obtain the fused position of the detected map element.

[0145] Among them, when performing position fusion to obtain the fusion position of the map element, the position average of the element positions corresponding to the same map element can be determined as the fusion position, or the element positions corresponding to the same map element can be weighted summed and the result determined as the fusion position, which is not limited in this embodiment of the present application.

[0146] It can be seen that the fusion position of the map element is obtained by fusion of the element positions corresponding to the same map element in the image collected by the visual sensor, that is, the positions of multiple elements corresponding to the same map element are comprehensively considered, and finally the fusion position of the map element is obtained. It can be seen that the above fusion position eliminates the randomness of the obtained single element position and can more accurately represent the actual position of the map element. Therefore, building a map for the road based on the obtained fusion position is conducive to reducing the error generated when building a map based on a single element position, and improving the accuracy of the constructed map.

[0147] As can be seen from the above, when the scheme provided by the embodiment of the present application is used to construct a map, the sensor data collected by the sensors carried by the vehicle during the driving process of the vehicle is obtained, and then SLAM can be performed based on the obtained sensor data to obtain reconstruction information, and the road surface description information of the road traveled by the vehicle can be constructed based on the reconstruction information, so that the element position of the map element in the image collected by the visual sensor in the road can be determined based on the road surface description information, and then the map for the road can be successfully constructed based on the determined element position. It can be seen that the above-mentioned map construction process can be realized based on the sensor data collected by the sensors carried by the vehicle, without the need for manual annotation by the staff, which reduces the workload of the map construction process and improves the efficiency of map construction.

[0148] In addition, in the solution provided in the embodiment of the present application, road surface description information is constructed based on the reconstruction information obtained by SLAM, and the road surface description information is used to accurately describe the road that the vehicle has traveled. Therefore, compared with the position of map elements in the road predicted based on experience, the element position of the detected map elements in the road can be more accurately determined based on the above-mentioned road surface description information, and thus a more accurate map can be constructed based on the determined element position.

[0149] Furthermore, when using the embodiment of the present application to construct a map, there is no need to pre-calibrate the position and posture of the sensors mounted on the vehicle, so there is no need to ensure that the position and posture of the sensors when collecting data are consistent with the position and posture during calibration, which simplifies the overall process of the map construction solution and improves the feasibility and flexibility of the solution.

[0150] Furthermore, when using the embodiments of the present application to build a map, it is possible to rely only on low-cost sensors such as monocular cameras, IMUs, and GPS sensors for SLAM, thereby building a map based on the results of SLAM, reducing the cost of the map building solution.

[0151] The following is an introduction to the method of performing SLAM to obtain reconstruction information based on the image collected by the visual sensor, the inertial information collected by the inertial sensor, and the positioning information collected by the positioning sensor in the sensor data mentioned in step S102.

[0152] Specifically, SLAM can be performed through the following steps E-I to obtain reconstruction information.

[0153] Step E: extracting visual feature points of the image collected by the visual sensor in the obtained sensor data, and matching the visual feature points of adjacent images.

[0154] The above-mentioned visual feature points can be SIFT (Scale Invariant Feature Transform) feature points, SURF (Speeded Up Robust Features) feature points, ORB (Oriented FAST and Rotated BRIEF) feature points, etc., which are not limited in the embodiments of the present application.

[0155] The adjacent images mentioned above refer to the images adjacent to each other at the corresponding image acquisition moment; matching the visual feature points of adjacent images refers to: for a certain image, determining the corresponding visual feature points of each visual feature point in the image in its adjacent image.

[0156] Step F: Based on the matching result, determine the first relative posture when the visual sensor captures adjacent images and the spatial position of the matched visual feature points in the visual sensor coordinate system.

[0157] Specifically, a brute force matching algorithm based on Hamming distance, a cross matching algorithm, a KNN (k-nearest neighbor classification) matching algorithm and other algorithms can be used to match visual feature points, so that the first relative pose when the visual sensor captures adjacent images and the spatial position of the matched visual feature points in the visual sensor coordinate system can be obtained based on the matching results.

[0158] Step G: Based on the inertial information collected by the inertial sensor in the obtained sensor data, determine the second relative posture when the visual sensor collects adjacent images.

[0159] In this step, the obtained inertial information can be input into an inertial model such as a strapdown inertial navigation model and a pre-integration model to solve for the pitch angle, roll angle, heading angle and other posture information of the visual sensor, thereby obtaining the second relative posture when the visual sensor collects adjacent images.

[0160] Step H: Based on the positioning information collected by the positioning sensor in the obtained sensor data, determine the global posture and driving speed of the vehicle when the visual sensor collects the image.

[0161] The above positioning information may include the absolute position, driving direction, driving speed, etc. of the vehicle. Based on the above positioning information, the global posture when the visual sensor captures the image and the driving speed of the vehicle can be determined.

[0162] Step I: Perform SLAM according to the first relative posture, the determined spatial position, the second relative posture, the global posture and the driving speed to obtain reconstruction information.

[0163] After obtaining the above-mentioned multiple posture information and speed information, SLAM can be performed on the above-mentioned information based on fusion algorithms such as factor graph algorithm and filter algorithm, and the obtained road feature points and / or vehicle posture are used as reconstruction information.

[0164] The above information involved in SLAM may be all the information obtained or part of the information obtained, which is not limited in the embodiments of the present application.

[0165] In this way, when performing SLAM, after obtaining the first relative pose and spatial position of the visual feature points when the visual sensor collects adjacent images based on the images collected by the visual sensor, on the first hand, the inertial information collected by the inertial sensor is taken into account, and the second relative pose when the visual sensor collects adjacent images is obtained. On the second hand, the positioning information collected by the positioning sensor is taken into account, and the global pose and the driving speed of the vehicle when the visual sensor collects the image are obtained. Based on the above two aspects of information, the first relative pose and the spatial position of the visual feature points obtained based on the image can be further constrained, so that SLAM can be performed according to the first relative pose, the determined spatial position, the second relative pose, the global pose and the driving speed, and more accurate reconstruction information can be obtained.

[0166] exist Figure 1 On the basis of the embodiment shown, when determining the element position of the detected map element in the road, the conversion relationship between the pixel position of the pixel point in the image and the spatial position in the real three-dimensional space can be obtained first, so that the projection ray of the pixel point describing the map element in the image in the real three-dimensional space can be determined based on the obtained conversion relationship, and then the element position of the above map element in the road can be determined based on the intersection of the projection ray and the road surface described by the road surface description information. In view of the above situation, the embodiment of the present application provides a second map construction method.

[0167] See also Figure 3 , is a flow chart of the first map construction method provided in an embodiment of the present application, the method includes the following steps S301-S309.

[0168] Step S301: Obtain sensor data collected by sensors mounted on the vehicle during the vehicle's driving process.

[0169] Step S302: Perform SLAM based on the acquired sensor data to obtain reconstruction information.

[0170] Step S303: performing map element detection on the image collected by the visual sensor in the obtained sensor data.

[0171] Step S304: Constructing road surface description information of the road traveled by the vehicle based on the reconstruction information.

[0172] The above steps S301 to S304 are the same as the above Figure 1 In the illustrated embodiment, steps S101 to S104 are the same and will not be described in detail here.

[0173] Step S305: Based on the sensor posture when the visual sensor collects the image and the sensor internal parameters of the visual sensor, a conversion relationship between the pixel position of the pixel point in the image collected by the visual sensor and the spatial position in the real three-dimensional space is obtained.

[0174] The embodiment of the present application does not limit the method for obtaining the above-mentioned sensor posture, and is explained below by way of example.

[0175] In one implementation, the vehicle pose of the vehicle may be determined based on data collected by the pose sensor, and the sensor pose of the visual sensor may be determined based on the obtained vehicle pose and the relative pose of the visual sensor in the vehicle.

[0176] In another implementation, the motion trajectory of the visual sensor may be estimated based on the visual odometer technology, and then the sensor pose of the visual sensor may be calculated according to the motion trajectory.

[0177] The sensor intrinsic parameters of the above-mentioned visual sensor are inherent properties of the visual sensor, and are generally fixed after the sensor leaves the factory. For example, if the above-mentioned visual sensor is a monocular camera, the sensor intrinsic parameters are the camera intrinsic parameter matrix of the monocular camera.

[0178] After obtaining the above-mentioned sensor pose and sensor intrinsic parameters, the conversion relationship between the pixel position of the pixel point in the image captured by the visual sensor and the spatial position in the real three-dimensional space can be obtained by reverse projection based on the sensor pose when the visual sensor captures the image and the sensor intrinsic parameters of the visual sensor.

[0179] Specifically, the conversion relationship between the pixel position of a pixel point in an image captured by a visual sensor and the spatial position in a real three-dimensional space can be obtained according to the following expression:

[0180]

[0181] Among them, (u, v) represents the pixel position, (x, y, z) represents the spatial position, T represents the sensor pose when the visual sensor acquires the image, K represents the sensor intrinsic parameter of the visual sensor, and d represents the depth of the pixel in the sensor coordinate system established with the viewpoint of the visual sensor as the origin.

[0182] Specifically, T can be represented by the following matrix: Among them, R represents the rotation parameter, and T represents the translation parameter;

[0183] K can be represented by the following matrix: Among them, f x and f y Respectively represent the length of the focal length in the X-axis and Y-axis directions, (c x , c y ) represents the principal point position of the vision sensor.

[0184] It can be seen that after obtaining the sensor intrinsic parameters and sensor posture, the back projection method can be used to conveniently and accurately obtain the conversion relationship between the pixel position of the pixel point in the image captured by the visual sensor and the spatial position in the real three-dimensional space.

[0185] Step S306: Based on the obtained conversion relationship, determine the projection ray of the pixel points describing the map elements in the image collected by the visual sensor in the real three-dimensional space.

[0186] The pixel points describing the map element may be all the pixel points describing the map element, or may be feature points such as the center point and corner point of the map element.

[0187] In this step, the pixel positions of the pixel points describing the map elements in the image collected by the visual sensor are substituted into the obtained conversion relationship, and the projection rays of the above pixel points in the real three-dimensional space can be obtained.

[0188] Specifically, it can be known from the above step S305 that the conversion relationship between the pixel position of the pixel point in the image collected by the visual sensor and the spatial position in the real three-dimensional space can be expressed by the following expression:

[0189]

[0190] Among them, d represents the depth of the pixel in the sensor coordinate system established with the viewpoint of the visual sensor as the origin, which is an unknown parameter. Therefore, by substituting the pixel position (u, v) of the pixel describing the map element in the image collected by the visual sensor into the above expression, the spatial ray represented by d can be solved, and the obtained spatial ray is the projection ray of the above pixel in the real three-dimensional space.

[0191] Step S307: Determine the intersection point of the projection ray and the road surface described by the road surface description information.

[0192] Since the projection ray is a ray in the real three-dimensional space, the intersection point of the projection ray and the road surface described by the road surface description information can be determined.

[0193] Combine the following Figure 4 , which provides a more intuitive explanation of the above intersection.

[0194] See also Figure 4 , is a schematic diagram of an intersection provided in an embodiment of the present application.

[0195] It can be seen that Figure 4 The figure shows the four projection rays of the pixels corresponding to the four corner points of the map element A in the image P in the real three-dimensional space. The four projection rays intersect with the road surface R to obtain Figure 4 The intersection shown in (solid circle in road surface R).

[0196] It needs to be said that Figure 4 The content shown is only a simple illustration for ease of understanding and is not strictly consistent with the actual situation.

[0197] Specifically, when the road surface description information is a road surface equation, the ray equation of the projection ray and the road surface equation can be combined, and the solution of the combined equation group is the intersection of the projection ray and the road surface described by the road surface description information.

[0198] Step S308: Based on the determined intersection point, determine the element position of the map element in the road in the image collected by the visual sensor.

[0199] The center point of the obtained intersection points, the average point of the obtained intersection points, etc. can be used as the element position.

[0200] Step S309: constructing a map for the road based on the determined element positions.

[0201] The above step 309 is the same as the above Figure 1 Step S106 in the illustrated embodiment is the same and will not be described again here.

[0202] As can be seen from the above, when the solution provided by the embodiment of the present application is used to determine the element position of a map element, first, based on the conversion relationship between the pixel position of the pixel point in the image collected by the visual sensor and the spatial position in the real three-dimensional space, the projection ray of the pixel point describing the map element in the above image in the real three-dimensional space is determined, and then based on the obtained road surface description information, the intersection of the projection ray and the road surface described by the road surface description information is determined. The position of the above intersection is the position of the pixel point describing the map element in the road surface, so that the element position of the map element in the road can be accurately obtained based on the above intersection. It can be seen that compared with the position of the map element in the road predicted based on experience, the accuracy of the determined element position is greatly improved, so the accuracy of constructing the map can be further improved based on the determined element position.

[0203] Corresponding to the above-mentioned map construction method, the embodiment of the present application also provides a map construction device.

[0204] See also Figure 5 , is a structural schematic diagram of a map construction device provided in an embodiment of the present application, and the above-mentioned device includes the following modules.

[0205] The sensor data acquisition module 501 is used to obtain sensor data collected by sensors carried by the vehicle during the vehicle's driving process, wherein the sensors carried by the vehicle include: visual sensors and posture sensors, wherein the posture sensors are: non-visual sensors used for simultaneous positioning and map reconstruction (SLAM);

[0206] A reconstruction information acquisition module 502 is used to perform SLAM based on the obtained sensor data, obtain reconstruction information, and perform map element detection on the image collected by the visual sensor in the obtained sensor data;

[0207] A road surface description information construction module 503, configured to construct road surface description information of the road traveled by the vehicle based on the reconstruction information;

[0208] An element position determination module 504, configured to determine the element position of the detected map element in the road based on the road description information;

[0209] The map construction module 505 is used to construct a map for the road based on the determined element positions.

[0210] As can be seen from the above, when the scheme provided in the embodiment of the present application is used to construct a map, the sensor data collected by the sensors carried on the vehicle during the driving process of the vehicle is obtained, and then SLAM can be performed based on the obtained sensor data to obtain reconstruction information, and the road surface description information of the road traveled by the vehicle can be constructed based on the reconstruction information, so that the element position of the map element in the image collected by the visual sensor in the road can be determined based on the road surface description information, and then the map for the road can be successfully constructed based on the determined element position. It can be seen that the above-mentioned map construction process can be realized based on the sensor data collected by the sensors carried on the vehicle, without the need for manual annotation by the staff, which reduces the workload of the map construction process and improves the efficiency of map construction.

[0211] In addition, in the solution provided in the embodiment of the present application, road surface description information is constructed based on the reconstruction information obtained by SLAM, and the road surface description information is used to accurately describe the road that the vehicle has traveled. Therefore, compared with the position of map elements in the road predicted based on experience, the element position of the detected map elements in the road can be more accurately determined based on the above-mentioned road surface description information, and thus a more accurate map can be constructed based on the determined element position.

[0212] Furthermore, when using the embodiment of the present application to construct a map, there is no need to pre-calibrate the position and posture of the sensors mounted on the vehicle, so there is no need to ensure that the position and posture of the sensors when collecting data are consistent with the position and posture during calibration, which simplifies the overall process of the map construction solution and improves the feasibility and flexibility of the solution.

[0213] In one embodiment of the present application, the reconstruction information includes: road surface feature points, and the road surface description information construction module 503 is specifically used to determine the road surface parameters of the road traveled by the vehicle based on at least three non-co-linear road surface feature points; based on the determined road surface parameters, generate the road surface equation of the road as the road surface description information of the road.

[0214] By selecting at least three non-collinear road feature points, the plane equation of the plane where the feature points are located can be obtained more accurately based on the coplanar constraints of the feature points. This plane equation is also the road surface equation of the road on which the vehicle travels. Therefore, this improves the accuracy of the determined road surface equation.

[0215] In addition, based on the road feature points, a parameterized estimate of the road surface on which the vehicle travels is performed, thereby obtaining a parameterized road surface equation, which is conducive to more accurate determination of the position of map elements in the road based on the road surface equation, thereby improving the accuracy of map construction.

[0216] In one embodiment of the present application, the reconstruction information includes: the vehicle posture of the vehicle, the road description information construction module 503, which is specifically used to determine the road surface normal vector of the road traveled by the vehicle based on the vehicle posture; determine the real road surface point in the road traveled by the vehicle based on the vehicle posture and the pre-obtained height of the vehicle relative to the road surface; determine the road surface intercept parameter of the road according to the road surface normal vector and the real road surface point; and generate the road surface equation of the road based on the road surface intercept parameter and the normal vector as the road surface description information road surface equation of the road.

[0217] It can be seen that the road surface normal vector of the road the vehicle has traveled on can be directly determined based on the vehicle posture, and then the road surface intercept parameters of the road the vehicle has traveled on can be conveniently solved by combining the above road surface normal vector and the real road surface points. Then, the road surface equation of the road the vehicle has traveled on can be conveniently obtained based on the road surface normal vector and the road surface intercept parameters, thereby improving the efficiency of determining the road surface equation.

[0218] Moreover, in the process of determining the above-mentioned road surface equation, there is no need to use the road surface feature points obtained based on SLAM. Since the road surface feature points obtained based on SLAM often have certain errors, this reduces the dependence on road surface feature points when determining the road surface equation, which is conducive to improving the accuracy of the determined road surface equation.

[0219] In addition, based on the vehicle posture, the road surface on which the vehicle travels is parameterized and estimated, thereby obtaining a parameterized road surface equation, which is conducive to more accurate determination of the position of map elements in the road based on the road surface equation, thereby improving the accuracy of map construction.

[0220] In one embodiment of the present application, the element position determination module 504 includes:

[0221] A conversion relationship acquisition submodule, used to obtain a conversion relationship between a pixel position of a pixel point in an image acquired by the visual sensor and a spatial position in a real three-dimensional space based on a sensor posture when the visual sensor acquires an image and a sensor internal parameter of the visual sensor;

[0222] A projection ray determination submodule, used to determine the projection ray of the pixel points describing the map elements in the image collected by the visual sensor in the real three-dimensional space based on the obtained conversion relationship;

[0223] An intersection determination submodule, used to determine the intersection of the projection ray and the road surface described by the road surface description information;

[0224] The element position determination submodule is used to determine the element position of the map element in the image collected by the visual sensor on the road based on the determined intersection point.

[0225] As can be seen from the above, when the solution provided by the embodiment of the present application is used to determine the element position of a map element, first, based on the conversion relationship between the pixel position of the pixel point in the image collected by the visual sensor and the spatial position in the real three-dimensional space, the projection ray of the pixel point describing the map element in the above image in the real three-dimensional space is determined, and then based on the obtained road surface description information, the intersection of the projection ray and the road surface described by the road surface description information is determined. The position of the above intersection is the position of the pixel point describing the map element in the road surface, so that the element position of the map element in the road can be accurately obtained based on the above intersection. It can be seen that compared with the position of the map element in the road predicted based on experience, the accuracy of the determined element position is greatly improved, so the accuracy of constructing the map can be further improved based on the determined element position.

[0226] In one embodiment of the present application, the conversion relationship acquisition submodule is specifically used to obtain the conversion relationship between the pixel position of a pixel point in the image captured by the visual sensor and the spatial position in the real three-dimensional space by reverse projection based on the sensor posture when the visual sensor captures the image and the sensor intrinsic parameters of the visual sensor.

[0227] It can be seen that after obtaining the sensor intrinsic parameters and sensor pose, based on the above expressions, the conversion relationship between the pixel position of the pixel point in the image captured by the visual sensor and the spatial position in the real three-dimensional space can be obtained conveniently and accurately.

[0228] In one embodiment of the present application, the posture sensor includes: an inertial sensor and a positioning sensor; the reconstruction information acquisition module 502 includes:

[0229] The reconstruction information acquisition submodule is used to perform SLAM based on the image collected by the visual sensor, the inertial information collected by the inertial sensor, and the positioning information collected by the positioning sensor in the sensor data to obtain reconstruction information.

[0230] In this way, on the basis of the images collected by the visual sensor, the inertial information collected by the inertial sensor and the positioning information collected by the positioning sensor are combined for SLAM. It can be seen that this not only takes into account the visual information, but also the inertial information and positioning information, so that more accurate reconstruction information can be obtained based on a variety of different types of information.

[0231] In one embodiment of the present application, the reconstruction information acquisition submodule is specifically used to extract visual feature points of the image captured by the visual sensor in the obtained sensor data, and match the visual feature points of adjacent images; based on the matching results, determine the first relative posture of the visual sensor when collecting the adjacent images and the spatial position of the matched visual feature points in the visual sensor coordinate system; based on the inertial information collected by the inertial sensor in the obtained sensor data, determine the second relative posture of the visual sensor when collecting the adjacent images; based on the positioning information collected by the positioning sensor in the obtained sensor data, determine the global posture of the visual sensor when collecting the image and the driving speed of the vehicle; perform SLAM according to the first relative posture, the determined spatial position, the second relative posture, the global posture and the driving speed to obtain reconstruction information.

[0232] In this way, when performing SLAM, after obtaining the first relative pose and spatial position of the visual feature points when the visual sensor collects adjacent images based on the images collected by the visual sensor, on the first hand, the inertial information collected by the inertial sensor is taken into account, and the second relative pose when the visual sensor collects adjacent images is obtained. On the second hand, the positioning information collected by the positioning sensor is taken into account, and the global pose and the driving speed of the vehicle when the visual sensor collects the image are obtained. Based on the above two aspects of information, the first relative pose and the spatial position of the visual feature points obtained based on the image can be further constrained, so that SLAM can be performed according to the first relative pose, the determined spatial position, the second relative pose, the global pose and the driving speed, and more accurate reconstruction information can be obtained.

[0233] In one embodiment of the present application, the map construction module 505 is specifically used to perform position fusion on the element positions corresponding to the same map element in the image collected by the visual sensor to obtain the fused position of the detected map element; and construct a map for the road based on the obtained fused position.

[0234] It can be seen that the fusion position of the map element is obtained by fusion of the element positions corresponding to the same map element in the image collected by the visual sensor, that is, the positions of multiple elements corresponding to the same map element are comprehensively considered, and finally the fusion position of the map element is obtained. It can be seen that the above fusion position eliminates the randomness of the obtained single element position and can more accurately represent the actual position of the map element. Therefore, building a map for the road based on the obtained fusion position is conducive to reducing the error generated when building a map based on a single element position, and improving the accuracy of the constructed map.

[0235] In one embodiment of the present application, the visual sensor includes: a monocular camera.

[0236] Since monocular cameras are relatively low in cost, using monocular cameras can save the cost of map construction solutions.

[0237] In the solutions provided in the embodiments of the present application, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are all carried out under the premise of the user's knowledge and authorization, and comply with the relevant laws and regulations and do not violate public order and good morals.

[0238] The present application also provides an electronic device, such as Figure 6 As shown, including:

[0239] Memory 601, used for storing computer programs;

[0240] The processor 602 is used to implement the above-mentioned map construction method when executing the program stored in the memory 601.

[0241] Furthermore, the electronic device may further include a communication bus and / or a communication interface, and the processor 602, the communication interface, and the memory 601 communicate with each other via the communication bus.

[0242] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0243] The communication interface is used for communication between the above electronic device and other devices.

[0244] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0245] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0246] In yet another embodiment provided in the present application, a vehicle is also provided, wherein the vehicle is equipped with the above-mentioned electronic device.

[0247] In another embodiment provided in the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned map construction method are implemented.

[0248] In another embodiment provided by the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any map construction method in the above embodiments.

[0249] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or other media (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0250] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0251] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, electronic device, storage medium and vehicle embodiments, since they are basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0252] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A map construction method, characterized in that: The method comprises: Obtaining sensor data collected by sensors carried on the vehicle during the vehicle's driving, wherein the sensors carried on the vehicle include: a visual sensor and a posture sensor, and the posture sensor is: a non-visual sensor used for simultaneous positioning and map reconstruction (SLAM); Performing SLAM based on the obtained sensor data to obtain reconstruction information, and performing map element detection on the image collected by the visual sensor in the obtained sensor data; Constructing road surface description information of the road traveled by the vehicle based on the reconstruction information; Based on the road surface description information, determining an element position of the detected map element in the road; Based on the determined element positions, a map for the road is constructed.

2. The method according to claim 1, characterized in that The reconstruction information includes: road feature points, and the road description information of the road traveled by the vehicle based on the reconstruction information includes: Determining road surface parameters of the road traveled by the vehicle based on at least three non-co-route road surface feature points; According to the determined road surface parameters, a road surface equation of the road is generated as road surface description information of the road.

3. The method according to claim 1, characterized in that The reconstruction information includes: the vehicle posture of the vehicle, and the road surface description information of the road traveled by the vehicle based on the reconstruction information includes: Based on the vehicle posture, determining a road surface normal vector of the road traveled by the vehicle; Determining a real road surface point on the road where the vehicle travels based on the vehicle posture and a pre-acquired height of the vehicle relative to the road surface; Determining a road surface intercept parameter of the road according to the road surface normal vector and a real road surface point; Based on the road surface intercept parameter and the normal vector, a road surface equation of the road is generated as road surface description information of the road.

4. The method according to claim 1, characterized in that The determining, based on the road surface description information, an element position of the detected map element in the road comprises: Based on the sensor posture when the visual sensor collects the image and the sensor internal parameters of the visual sensor, a conversion relationship between the pixel position of the pixel point in the image collected by the visual sensor and the spatial position in the real three-dimensional space is obtained; Based on the obtained conversion relationship, determining the projection ray of the pixel point describing the map element in the image collected by the visual sensor in the real three-dimensional space; Determine an intersection point between the projection ray and the road surface described by the road surface description information; Based on the determined intersection point, an element position of a map element in the road in the image collected by the visual sensor is determined.

5. The method according to claim 4, characterized in that The step of obtaining a conversion relationship between a pixel position of a pixel point in an image collected by the visual sensor and a spatial position in a real three-dimensional space based on the sensor posture when the visual sensor collects the image and the sensor internal parameter of the visual sensor comprises: Based on the sensor posture when the visual sensor collects images and the sensor internal parameters of the visual sensor, the conversion relationship between the pixel position of the pixel point in the image collected by the visual sensor and the spatial position in the real three-dimensional space is obtained by back projection.

6. The method according to claim 1, characterized in that The posture sensor includes: an inertial sensor and a positioning sensor; The step of performing SLAM based on the obtained sensor data to obtain reconstruction information includes: SLAM is performed based on the image collected by the visual sensor, the inertial information collected by the inertial sensor, and the positioning information collected by the positioning sensor in the sensor data to obtain reconstruction information.

7. The method according to claim 6, characterized in that The step of performing SLAM based on the image collected by the visual sensor, the inertial information collected by the inertial sensor, and the positioning information collected by the positioning sensor in the sensor data to obtain reconstruction information includes: Extracting visual feature points of the image collected by the visual sensor in the obtained sensor data, and matching the visual feature points of adjacent images; Based on the matching result, determining a first relative posture when the visual sensor acquires the adjacent images and a spatial position of the matched visual feature point in the coordinate system of the visual sensor; Determine, based on the inertial information collected by the inertial sensor in the obtained sensor data, a second relative posture when the visual sensor collects the adjacent images; Determine the global position of the visual sensor when acquiring images and the driving speed of the vehicle based on the positioning information collected by the positioning sensor in the obtained sensor data; SLAM is performed according to the first relative posture, the determined spatial position, the second relative posture, the global posture and the driving speed to obtain reconstruction information.

8. The method according to any one of claims 1 to 7, characterized in that The step of constructing a map for the road based on the determined element positions comprises: Performing position fusion on element positions corresponding to the same map element in the image collected by the visual sensor to obtain a fused position of the detected map element; Based on the obtained fused positions, a map for the road is constructed.

9. A map construction device, characterized in that: The device comprises: A sensor data acquisition module, used to obtain sensor data collected by sensors carried on the vehicle during the vehicle's driving process, wherein the sensors carried on the vehicle include: visual sensors and posture sensors, and the posture sensors are: non-visual sensors used for simultaneous positioning and map reconstruction SLAM; A reconstruction information acquisition module is used to perform SLAM based on the obtained sensor data to obtain reconstruction information, and to perform map element detection on the image collected by the visual sensor in the obtained sensor data; A road surface description information construction module, used to construct road surface description information of the road traveled by the vehicle based on the reconstruction information; An element position determination module, used to determine the element position of the detected map element in the road based on the road description information; A map building module is used to build a map for the road based on the determined element positions.

10. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, for implementing any of the methods described in claims 1-8 when executing a program stored in a memory.

11. A vehicle, characterized in that: The vehicle is equipped with the electronic device according to claim 10.

Images