A method, device and equipment for constructing point cloud map

Abstract

Disclosed is a method, device and equipment for constructing a point cloud map, the method comprising: obtaining feature point information of multiple frames of images, first speed information of a vehicle and positioning information of the vehicle, wherein the feature point information comprises: feature points of each frame of images and matching relationships between feature points between frames, the first speed information of the vehicle comprises the first linear speed and / or first angular speed of the vehicle, the first speed information is measured by a wheel speed sensor; determining the first posture information of the vehicle according to the feature point information and the first speed information; determining the second posture information of the vehicle according to the first posture information and the positioning information of the vehicle; and reconstructing a point cloud using the feature point information and the second posture information of multiple frames of images to obtain a point cloud map. The method obtains the linear speed and / or angular speed of the vehicle by measuring the wheel speed sensor, replacing the IMU to measure the speed information of the vehicle, thereby avoiding the introduction of noise by the IMU measurement, and avoiding the installation and arrangement of the IMU, saving production costs.

Description

Technical Field

[0001] The present application relates to the field of computer vision and point cloud map construction, and in particular to a point cloud map construction method, device and equipment. Background Art

[0002] For autonomous driving, the acquisition of positioning information is crucial. Accurately positioning the current position of the vehicle and the objects in the surrounding environment of the vehicle, and simulating the roads and surrounding environment on both sides of the vehicle can improve the accuracy of autonomous driving. When mapping the roads on both sides of the vehicle, sensors on the vehicle, such as cameras and inertial measurement units (IMUs), will collect real-time information about the vehicle's surroundings, such as the vehicle's speed, surrounding moving objects, pedestrians, trees, grass, etc., and then send the collected information to the server.

[0003] The server will build a map of the vehicle's surroundings based on the measurement data reported by each sensor. However, the reported raw data is easily affected by the surrounding environment. For example, it may be affected by noise, or be blocked by buildings or natural objects, which may result in inaccurate positioning of the vehicle. As a result, when the inaccurate vehicle position is used to build a map, the matching information lacks robustness. Summary of the invention

[0004] In order to solve the problems caused by the above-mentioned visual sensor mapping, the present application is proposed. The embodiments of the present application provide a point cloud map construction method and device for solving the drawbacks of pure visual mapping and multi-sensor mapping.

[0005] Specifically, the embodiments of the present application provide the following technical solutions:

[0006] In a first aspect, the present disclosure provides a method for constructing a point cloud map, which can be implemented by a network device, such as a server, or a terminal device, such as a PC, and includes:

[0007] Acquire feature point information of multiple frames of images, first speed information of the vehicle, and positioning information of the vehicle, determine the first posture information of the vehicle based on the feature point information and the first speed information; determine the second posture information of the vehicle based on the first posture information and the positioning information of the vehicle; and perform point cloud reconstruction using the feature point information of the multiple frames of images and the second posture information to obtain a point cloud map.

[0008] The feature point information includes: the feature points of each frame image and the matching relationship between the feature points between frames, the first speed information of the vehicle includes the first linear speed and / or first angular speed of the vehicle, and the first speed information is measured by a wheel speed sensor. The positioning information of the vehicle includes the GPS position information of the vehicle, and the GPS position information has a physical meaning in the vertical direction.

[0009] The method provided in this aspect obtains the linear speed and / or angular speed of the vehicle by measuring the wheel speed sensor, replacing the IMU to measure the speed information of the vehicle, thereby avoiding the introduction of noise by the IMU measurement, and at the same time avoiding the installation and layout of different types of IMUs, saving production costs.

[0010] In addition, the linear speed and / or angular velocity of the vehicle measured by the wheel speed sensor or tachometer is combined with the feature point matching relationship to obtain accurate first pose information of the vehicle, and then fused with the vehicle positioning information to obtain second pose information of the vehicle. This allows the vehicle to increase the scale significance of the vertical direction of the positioning information on the basis of the original accurate camera pose, thereby obtaining second pose information with three-dimensional features, which can provide robust input for other calculations and development of Ground Truth maps.

[0011] In a second aspect disclosed in the present application, a point cloud map construction device is provided, the device comprising: an acquisition module, a calculation module and a composition module, wherein:

[0012] The acquisition module is used to acquire feature point information of multiple frames of images, first speed information of the vehicle and positioning information of the vehicle, wherein the feature point information includes: feature points of each frame of image and matching relationship between feature points between frames, the first speed information of the vehicle includes a first linear speed and / or a first angular speed of the vehicle, and the first speed information is measured by a wheel speed sensor;

[0013] The calculation module is used to determine the first posture information of the vehicle according to the feature point information and the first speed information obtained from the acquisition module, and determine the second posture information of the vehicle according to the first posture information and the positioning information of the vehicle obtained by the acquisition module;

[0014] The composition module is used to reconstruct the point cloud using the feature point information of multiple frames of images and the second posture information obtained by the calculation module to obtain a point cloud map.

[0015] The third aspect disclosed in the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed, the point cloud map construction method described in the first aspect is implemented.

[0016] The fourth aspect disclosed in the present application also provides an electronic device, which includes: a processor and a memory, the memory being used to store executable instructions; the processor being used to read the executable instructions from the memory and execute the instructions to implement the point cloud map construction method described in the first aspect above.

[0017] Optionally, the electronic device is a network device, such as a server or a controller.

[0018] The fifth aspect disclosed in the present application also provides a point cloud map construction system, which includes a server and at least one client. The at least one client can be any terminal device, such as a user equipment UE, or the client can also be a vehicle computer. Furthermore, taking the vehicle computer as an example, the vehicle computer is provided with multiple sensors, such as including but not limited to cameras / camcorders, wheel speed sensors / speedometers, and GPS sensors.

[0019] The vehicle computer is used to obtain multiple frames of images taken by the camera, vehicle speed information measured by the wheel speed sensor, and vehicle positioning information measured by the GPS sensor, such as the vehicle's GPS position information, and send this information to the server.

[0020] The server is used to receive various information from the vehicle computer, and determine the first posture information of the vehicle based on the feature point information and the first speed information; determine the second posture information of the vehicle based on the first posture information and the positioning information of the vehicle; and use the feature point information of multiple frames of images and the second posture information to reconstruct point cloud to obtain a point cloud map.

[0021] Optionally, the server includes a display or a screen for displaying the point cloud map. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0023] Figure 1 It is a schematic diagram of a scenario structure provided in an embodiment of the present application.

[0024] Figure 2 It is a structural schematic diagram of a vehicle computer provided in an embodiment of the present application.

[0025] Figure 3 It is a structural diagram of a server provided in an embodiment of the present application.

[0026] Figure 4 It is a flow chart of a point cloud map construction method provided in an embodiment of the present application.

[0027] Figure 5 This is a schematic diagram of obtaining information reported by a sensor provided in an embodiment of the present application.

[0028] Figure 6 It is a schematic diagram of a method for determining the first position posture information of a vehicle provided in an embodiment of the present application.

[0029] Figure 7 It is a flow chart of a method for determining the second posture information of a vehicle provided in an embodiment of the present application.

[0030] Figure 8 It is a structural schematic diagram of a map construction device provided in an embodiment of the present application.

[0031] Fig. 9 It is a structural schematic diagram of another map construction device provided in an embodiment of the present application.

[0032] Fig.10 It is a structural schematic diagram of another vehicle computer provided in an embodiment of the present application. DETAILED DESCRIPTION

[0033] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0034] Application Overview

[0035] At present, the methods of single-shot mapping based on visual sensors mainly include pure visual mapping and multi-sensor mapping. Among them, the pure visual mapping method is to optimize the vehicle posture by simply extracting visual information, and then use visual feature points, lines and other information to construct cloud maps. The multi-sensor mapping method uses IMU to fuse with visual information, uses IMU pre-integration to obtain the vehicle's posture information, and then optimizes the posture information and feature points based on reprojection errors to obtain the vehicle's accurate posture information.

[0036] However, both of the above methods will have certain defects. For example, in the pure visual single-shot mapping method, the simple visual information extraction lacks scale information, resulting in the lack of physical meaning in the mapping result, and it cannot be used as the GroundTruth map to provide robust input for other subsequent computer vision research and development solutions. In addition, if a multi-sensor mapping method is adopted, the use of IMU for sensor fusion will introduce noise interference, reducing the accuracy of the vehicle's posture. Moreover, different models of IMU need to deploy different strategies during the mapping process, which will lead to increased development costs. The embodiments of the present application are intended to solve the above-mentioned technical problems. First, the technical scenarios involved in the technical solutions of the present application are introduced.

[0037] This application is applied to point cloud map construction scenarios, specifically to solve the drawbacks of pure visual mapping and multi-sensor mapping. For example, in the process of pure visual single-shot mapping, the simple visual information extraction lacks scale information, resulting in the mapping result having no physical meaning and being unable to be used as a label (Ground Truth) map to provide robust input for subsequent other computer vision research and development solutions. When using multi-sensor mapping, using IMU for sensor fusion will introduce noise, reduce the accuracy of the vehicle's posture, and increase the cost of IMU deployment.

[0038] Among them, "Ground truth" refers to the accuracy of the classification of the training set for supervised learning technology during the machine learning process. It is used to prove or disprove research hypotheses in statistical models. In this embodiment, "Ground truth" refers to the process of collecting appropriate target data for point cloud map construction.

[0039] In the traditional vehicle positioning and road construction process, IMU is used to determine the vehicle's speed information, including linear speed and / or angular speed. The solution disclosed in the present invention uses a wheel speed sensor or a wheel speed meter instead of the IMU to measure the vehicle's speed information, thereby solving the problems of high IMU deployment cost, large error and low positioning accuracy.

[0040] Exemplary Systems

[0041] See also Figure 1 , is a schematic diagram of a scenario structure provided in an embodiment of the present application. The system includes: a server 100 and at least one mobile terminal, wherein the at least one mobile terminal may be a vehicle-mounted terminal or a vehicle computer, such as a vehicle computer 200, wherein the vehicle computer 200 is connected to the server 100 via a wireless network, such as communicating with the server 100 via a base station 300.

[0042] The server 100 is a network device. Optionally, the network device may also be a controller, a data center, a cloud platform, etc.

[0043] The wireless network can be any wireless communication system, such as a Long Term Evolution (LTE) system or a fifth generation mobile communication system (5G). In addition, it can also be applied to later communication systems, such as sixth generation and seventh generation mobile communication systems.

[0044] The base station 300 includes but is not limited to (base station, BS) or (Base Transceiver Station, BTS). Further, the base station can be a base station (base transceiver station, BTS) in the global system for mobile communication (global system for mobile communication, GSM) or code division multiple access (code division multiple access, CDMA), and can also be an evolutionary base station (eNB / e-NodeB) in LTE, or an evolutionary base station (next generation eNB, ng-eNB) in the next generation LTE, or a base station (gNB) in NR.

[0045] The mobile terminal includes any type of mobile device or mobile apparatus, such as a vehicle, an aircraft, a spacecraft, etc., or a computing device with a mobile terminal. Figure 2 As shown, in one example, the vehicle computer 200 includes a communication interface 201 , a processor 202 and a memory 203 .

[0046] In addition, various types of sensors are also provided in the vehicle computer 200 for acquiring sample data of the current mobile environment in which the mobile terminal is moving, such as an image sensor 204 , a wheel speed sensor 205 , and a GPS sensor 206 .

[0047] Furthermore, the image sensor 204 is used to capture image data, such as obtaining multiple frames of images taken by the vehicle during driving, and the image sensor 204 can be a camera or a camera array. The wheel speed sensor 205, also known as a "wheel speed meter", is a sensor used to measure the rotation speed of the vehicle wheel. Commonly used wheel speed sensors mainly include: magnetoelectric wheel speed sensors and Hall wheel speed sensors. The GPS sensor 206 is used to obtain real-time positioning information of the vehicle (including GPS location information), such as the latitude and longitude and driving trajectory of the vehicle-mounted terminal or vehicle, and report the positioning information to the server 100. It should be understood that the above-mentioned various types of sensors can also include laser sensors and other sensors for capturing scan data, and this embodiment does not limit this.

[0048] The communication interface 201 is used to realize data transmission between the vehicle computer 200 and the server 100.

[0049] The memory 203 is used to store relevant data, such as image data, GPS location data and vehicle speed information.

[0050] In addition, if Figure 3 As shown, a structural diagram of a server 100 is also provided, including a communication interface 301, a processor 302, a memory 303, and an input / output device 304, etc.

[0051] Among them, the communication interface 301 is used to realize communication transmission with the vehicle computer 200, the processor 302 is used to implement the method described in the present disclosure, and the memory 303 is used to store relevant data and computer program instructions, etc. The input / output devices include a display, a mouse, a keyboard, a speaker, etc.

[0052] In addition, the server 100 may also include other devices, such as various sensors, communication interfaces, transceiver devices, etc. The embodiments of the present application do not limit the specific technologies and specific device forms used by the server and the client.

[0053] In addition, the technical solution provided in this embodiment can be implemented by software, hardware, or a combination of software and hardware. Among them, the hardware can provide image input, speed measurement of wheel speed meter / wheel speed sensor, and GNSS (Global Navigation Satellite System) input; the software can be implemented by C++ programming language, Java, etc., and the point cloud map construction function can be developed and implemented by Python programming voice, or it can also be implemented by other software and hardware. This application does not limit the specific hardware, software structure, and function of the implementation.

[0054] Exemplary Methods

[0055] See also Figure 4 , is a flow chart of a method for constructing a point cloud map provided by an exemplary embodiment of the present application. This embodiment can be applied to the above-mentioned server 100, and the method includes the following steps:

[0056] Step 110: Acquire feature point information of multiple frames of images, first speed information of the vehicle, and positioning information of the vehicle.

[0057] The feature point information includes: the feature points of each frame image and the matching relationship between the feature points between frames. The first speed information of the vehicle includes the first linear speed and / or first angular speed of the vehicle, and the first speed information is measured by a wheel speed sensor. The positioning information of the vehicle includes the GPS information of the vehicle, and the GPS position information includes the coordinate position of the vehicle in the GPS coordinate system, as well as the longitude and latitude information.

[0058] In addition, when the camera captures each frame of an image, each frame of the image also corresponds to a timestamp. For example, the timestamp corresponding to the first frame of the image is the first moment "t1", the timestamp corresponding to the second frame of the image is the second moment "t2", and so on. The shooting time interval can be set by the camera itself. The camera can report the image to the server after each frame of the image is captured, or report it to the server after a group of images are taken. The content captured in the photo is the surrounding environment captured by the camera during the vehicle's driving.

[0059] Optionally, in one implementation, the camera only reports the image (photo) to the server, and the server parses the acquired image (photo) to obtain the feature point information; or, in another implementation, the camera or the vehicle computer parses the captured image (photo), including performing relevant preprocessing on the image, detecting feature information such as points, lines, intervals, etc., and matching the previous and next key frames to obtain the feature point information of each frame image, and sending the feature point information of all images to the server.

[0060] In step 110, the speed information of the vehicle includes a first linear velocity and / or a first angular velocity, which can be measured and calculated by a wheel speed sensor or a wheel speed meter, and the speed information of the vehicle corresponds to a timestamp, that is, the first linear velocity and the first angular velocity are instantaneous velocities, and the timestamp is consistent with the timestamps of the first moment t1 and the second moment t2 captured above. The wheel speed meter / wheel speed sensor can determine the first linear velocity and the first angular velocity of the vehicle according to parameters such as the wheel speed, wheel diameter and / or wheel hub diameter of the vehicle.

[0061] It should be understood that the first linear velocity and / or the first angular velocity of the vehicle may be calculated by the vehicle computer 200 , or may be determined by a wheel speed meter or a wheel speed sensor, and then reported to the server 100 .

[0062] In addition, the vehicle computer 200 can report the feature point information of the image, the speed information and the positioning information of the vehicle at the same time, or report them to the server 100 in time periods or one by one. This embodiment does not limit the reporting time and the reporting time sequence.

[0063] Step 120: Determine the first posture information of the vehicle according to the feature point information and the first speed information.

[0064] Wherein, the first posture information describes the feature point set and posture information on the two-dimensional image. Furthermore, the posture represents the position and attitude. Any rigid body can be accurately and uniquely represented by position and attitude in the spatial coordinate system (OXYZ). Wherein, the position can be represented by: x, y, z coordinates, and the attitude can be represented by: the angle rx between the rigid body and the OX axis, the angle ry between the rigid body and the OY axis, and the angle rz between the rigid body and the OZ axis.

[0065] Step 130: Determine second posture information of the vehicle based on the first posture information and the positioning information of the vehicle.

[0066] Specifically, it includes: firstly aligning the first posture information and the positioning information of the vehicle to obtain the first posture information and the position information of the vehicle in the same coordinate system (such as the GPS coordinate system); then, optimizing the first posture information and the posture information of the vehicle in the same coordinate system to obtain the second posture information, wherein the second posture information is aligned with the GPS position information of the vehicle in the GPS coordinate system.

[0067] Optionally, the fitting method may adopt a loose coupling method.

[0068] The second position and posture information of the vehicle has a geographical meaning on a plane and also has a scale meaning in a vertical direction. The second position and posture information is a feature point set and position and posture information on a three-dimensional image.

[0069] Step 140: Reconstruct point cloud using feature point information of the multiple frames of images and the second pose information to obtain a point cloud map.

[0070] One possible implementation method is to reconstruct a dense point cloud map based on the obtained second pose information and the feature point information of the image obtained in the above step 110, that is, the sparse point cloud information, and perform subsequent aggregation and iterative operations on the sparse point cloud in the effective area to obtain a single-vision point cloud map, which is used to describe the surrounding environment of the vehicle.

[0071] Furthermore, the point cloud map reconstruction can adopt a multi-view system (MVS) method to obtain a single-view point cloud map by fusing the point cloud of the scene, thereby improving the accuracy of point cloud map reconstruction.

[0072] The method provided in this embodiment obtains the linear speed and / or angular velocity of the vehicle by measuring the wheel speed sensor, replacing the IMU to measure the vehicle speed information, thereby avoiding the introduction of noise by the IMU measurement and avoiding the installation and layout of different types of IMUs. The use of wheel speed sensors or wheel speed meters to measure the speed of the mobile terminal can save production costs.

[0073] In addition, the linear speed and / or angular velocity of the vehicle measured by the wheel speed sensor or tachometer is combined with the feature point matching relationship to obtain accurate first pose information of the vehicle, and then fused with the vehicle positioning information to obtain second pose information of the vehicle. On the basis of the original accurate camera pose information, the vehicle adds vertical scale meaning, thereby obtaining second pose information with three-dimensional features, providing robust input for other visual calculations and research and development of Ground Truth maps.

[0074] In the above Figure 4 Based on the embodiment shown, Figure 5 As shown, obtaining feature point information of multiple frames of images in step 110 may include the following steps:

[0075] Step 110 - 1 : extracting feature points from the multiple frames of images taken during the driving of the vehicle to obtain at least one feature point for each frame of image.

[0076] Step 110 - 2 : performing feature point matching based on at least one feature point of each frame image to obtain a matching relationship between feature points of the images between the frames.

[0077] In the process of extracting feature points, one or more of the following algorithms may be used: Deep Learning, ORB (Oriented FAST and Rotated BRIEF) feature point extraction algorithm, SHITOMASHI. In the process of matching the feature points of at least one extracted feature point, the optical flow method or ORB matching algorithm may be used to obtain the matching relationship of the feature points of the inter-frame images.

[0078] Taking the ORB algorithm as an example, the feature detection and extraction of the ORB algorithm is the process of obtaining key objects in images or videos transmitted by visual sensors. Combined with the extracted features, target matching based on feature points is performed to obtain the mapping relationship between feature points of each frame image. The calculation speed of the ORB algorithm is faster than SIFT, and it has good rotation invariance and noise resistance.

[0079] This embodiment does not limit the specific implementation process of image feature point extraction and feature point matching.

[0080] In step 110, in order to further improve the accuracy of the first posture information, the reported vehicle speed information is optimized in advance to obtain the first speed information, that is, the first speed information is the optimized linear speed / angular speed. Figure 5 As shown, the specific steps include:

[0081] 110-3: Obtain second speed information of the vehicle reported by the wheel speed sensor, where the second speed information includes a second linear speed and / or a second angular speed of the vehicle. The second linear speed and / or the second angular speed is the speed information of the vehicle directly measured by the wheel speed sensor, that is, the original measurement data.

[0082] 110 - 4 : Optimize the second speed information to obtain the first speed information.

[0083] Among them, the step 110-4 specifically includes: first pre-integrating the second speed information to obtain the position and posture information of the vehicle; then processing the second linear velocity and / or second angular velocity of the vehicle according to the position and posture information of the vehicle to obtain the first speed information.

[0084] Furthermore, a pre-integration process is derived as follows:

[0085] Assume that the second linear velocity of the vehicle at time t measured by the wheel speed meter or wheel speed meter sensor is The second angular velocity is in, Affected by white noise η and sensor bias b, Affected by the scale information, the relationship between the measured value and the true value can be expressed by equations (1) and (2):

[0086]

[0087]

[0088] Among them, ω(t) and v(t) represent theoretical values, and Indicates the actual measured value.

[0089] By simply transforming the above equations (1) and (2), we can obtain equations (3) and (4), which are expressed as:

[0090]

[0091]

[0092] The rotation matrix is ​​obtained according to the dynamic equation of the CAN bus. For example, equation (5) is a rotational dynamic model representing the rotation matrix:

[0093]

[0094] The rotation angle The Jacobian matrix (denoted as “J r ()”) is transformed, for example, equation (6) expresses the rotation angle by the angular axis method for:

[0095]

[0096] In addition, the dynamic equation of the vehicle position p(t) can be expressed as equation (7),

[0097]

[0098] Based on the above equations (5) to (7), in the above step 110-4, the second speed information is pre-integrated to obtain the position and posture information of the vehicle. The frequency of CAN measurement is synchronized with the camera frequency and timestamp. Assuming that the indexes of the two frames are i and j, the limited CAN measurement values ​​between the two frames are pre-integrated, and the following equations (8), (9) and (10) can be obtained, which are respectively expressed as:

[0099]

[0100]

[0101]

[0102] Among them, the position of the vehicle obtained after pre-integration processing is p j , the position p j and the position p of the i-th frame i and linear velocity v i 、v j The position of the vehicle is then determined by the rotation angle θ j Indicates that v j Represents the linear velocity of the vehicle in the i-th frame in the rotation matrix.

[0103] This embodiment is based on the posture measurement of multi-feature point matching. In this implementation, the 2D-3D matching relationship of the feature points is randomly sampled and solved, and the wrong matching point pairs are continuously tested and eliminated. Then, all the correct matching point pairs are used to re-solve the posture parameters of the target, and then the initial posture of the vehicle is obtained. Finally, the initial posture of the vehicle is used to optimize the original measured linear velocity / angular velocity to obtain the parameters after state update, including the Jacobian matrix, covariance matrix, etc., as well as the optimized linear velocity and angular velocity, that is, the first velocity information.

[0104] Optionally, an optimization method is to use LM (Levenberg-Marquardt) algorithm to optimize and obtain the first speed information. The LM algorithm is one of the optimization algorithms, and its purpose is to find a parameter vector that minimizes the function value, so as to obtain an accurate vehicle linear speed and / or angular speed.

[0105] In vector analysis, the Jacobian matrix is ​​a matrix in which first-order partial derivatives are arranged in a certain way. The importance of the Jacobian matrix lies in that it reflects the optimal linear approximation of a differentiable equation to a given point, so the Jacobian matrix is ​​similar to the derivative of a multivariate function. Assume that F:Rn→Rm is a function that transforms from Euclidean n-dimensional space to Euclidean m-dimensional space. This function F consists of m real functions: y1(x1,…,xn),…,ym(x1,…,xn). The partial derivatives of these functions can form a matrix with m rows and n columns, which is the Jacobian matrix. The Jacobian matrix is ​​used for the mapping operation of images between frames when the coordinate system is transformed, and is a key parameter for determining the pose information.

[0106] In another embodiment, when the second linear velocity and / or second angular velocity of the vehicle is processed according to the position and posture information of the vehicle in step 110-4, a first Jacobian matrix is ​​also obtained. Step 120 specifically includes: optimizing the feature point reprojection errors based on the feature points of the image, the matching relationship between the feature points of each frame image, and the first linear velocity and / or the first angular velocity to obtain the optimized first pose information, which has a physical scale meaning.

[0107] Furthermore, if Figure 6 As shown, the method includes:

[0108] 120-1: Perform a first reprojection error calculation on the feature point information to obtain a second Jacobian matrix.

[0109] Specifically, a first reprojection error calculation is performed on the feature point information based on each feature point to obtain the second Jacobian matrix.

[0110] 120-2: Update the second Jacobian matrix according to the first Jacobian matrix to obtain a third Jacobian matrix. In this step, the second Jacobian matrix is ​​optimized to obtain a third Jacobian matrix.

[0111] 120-3: Use the third Jacobian matrix to perform a second reprojection error calculation to obtain the first pose information.

[0112] Specifically, similar to step 120 - 1 , the third Jacobian matrix is ​​used to perform a second reprojection error calculation based on each feature point to obtain the first pose information.

[0113] In this embodiment, the first, second and third Jacobian matrices are mainly used to derive the position and rotation of the vehicle, as well as the derivatives of bias and noise with respect to the position and rotation, and have actual physical meanings such as linear velocity and angular velocity.

[0114] In addition, the first reprojection error and the second reprojection error can optimize the pose and three-dimensional feature points at the same time. The reprojection error is used when optimizing the camera motion pose (3D-2D) in SLAM (Simultaneous Localization And Mapping): it is the error obtained by comparing the pixel coordinates (observed projected position) with the position obtained by projecting the 3D point according to the current estimated pose.

[0115] Reprojection error is often used in computer vision. For example, when calculating the homography matrix and projection matrix of a plane, the reprojection error is often used to construct a cost function, and then the cost function is minimized to optimize the homography matrix or the projection matrix. This embodiment uses the reprojection error to calculate the Jacobian matrix, thereby optimizing the camera pose information, reducing the calculation error of the homography matrix and the measurement error of the image feature points, and making the measurement accuracy higher.

[0116] In addition, in this embodiment, when optimizing the speed information of the vehicle reported by the wheel speed sensor / wheel speed meter, the feature point information of the image is also optimized. Specifically, in the above step 110-4, the method further includes:

[0117] The server determines whether the first reprojection error (error1) is within a preset threshold value based on the position and posture information of the vehicle; if yes, the aforementioned step 120-3 is executed, and the second reprojection error (error2) based on the feature points is calculated using the third Jacobian matrix, and the visual information of the image is optimized to obtain the first posture information and visual optimization parameters. If not, the current frame is skipped and the next frame image is queried. If the next frame image is still inaccurate, the repositioning method is tried to supplement the posture information of the vehicle to obtain accurate posture information of the vehicle.

[0118] If the effects of vision optimization and tachometer optimization (the above steps 110-3 and 110-4) are good, that is, error1 is within the preset threshold value, then the estimation result is better, and the vision optimization parameters (for example, including error1) and the tachometer optimization parameters (for example, including scale factors, sensor bias parameters, etc.) are jointly optimized to obtain the optimized current frame trajectory information, which is the first posture information.

[0119] The following is a detailed description of step 130 in the aforementioned method embodiment.

[0120] The aforementioned step 130: determining the second posture information of the vehicle according to the first posture information and the positioning information of the vehicle, wherein the positioning information of the vehicle includes GPS location information, such as Figure 7 As shown, specifically including:

[0121] 130 - 1 : Fitting the first posture information and the GPS position information to obtain the first posture information and the GPS position information of the vehicle in a GPS coordinate system.

[0122] Before fitting, the first pose information is the position and attitude of the vehicle expressed in the camera coordinate system. After fitting, the first pose information is converted into the position and attitude expressed in the GPS coordinate system, thereby preparing for fitting the GPS position information.

[0123] 130-2: Perform pose graph optimization on the first pose information in the GPS coordinate system and the GPS position information of the vehicle to obtain the second pose information, where the second pose information is fitted with the GPS position information of the vehicle in the GPS coordinate system.

[0124] The "measurement" of the pose graph refers to the pose of each frame obtained by local optimization, and then the relative pose between adjacent key frames is calculated. In other words, the pose graph optimization uses the pose of each key frame as a node, and the relative pose calculation result between the previous key frames (the time interval should be short) as the measurement for maximum likelihood estimation. This method can reduce the calculation scale and obtain the second pose information of the camera. This pose information is the vehicle pose of a single mapping after global optimization and robust scale, providing accurate data for the point cloud map construction in step 140.

[0125] The method provided in this embodiment performs graph optimization processing on the posture (first posture information) output by sensor fusion optimization and the GPS location information with geographic information to obtain vehicle posture information (second posture information) with precise scale and a point cloud set of each key frame. Finally, the point cloud set is used to construct a point cloud map to obtain a point cloud map with higher accuracy and better robustness input.

[0126] Exemplary Devices

[0127] See also Figure 8 , is a map construction device provided in an embodiment of the present application, which may be a server or a functional module set on a server, and is used to implement all or part of the functions of the aforementioned method embodiment. Specifically, the map construction device includes: an acquisition module 801, a calculation module 802, and a composition module 803. In addition, the device may also include other more modules and units, such as a storage module, a sending module, etc., which are not limited in this embodiment.

[0128] Specifically, the acquisition module 801 is used to acquire feature point information of multiple frames of images, first speed information of the vehicle, and positioning information of the vehicle.

[0129] The feature point information includes: the feature points of each frame image and the matching relationship between the feature points between frames, the first speed information of the vehicle includes the first linear speed and / or first angular speed of the vehicle, and the first speed information is measured by a wheel speed sensor. The positioning information includes the GPS position information of the vehicle.

[0130] The calculation module 802 is used to determine the first posture information of the vehicle based on the feature point information and the first speed information obtained from the acquisition module 801, and to determine the second posture information of the vehicle based on the first posture information and the positioning information of the vehicle obtained by the acquisition module.

[0131] The composition module 803 is used to reconstruct the point cloud using the feature point information of the multiple frames of images and the second posture information obtained by the calculation module 802 to obtain a point cloud map.

[0132] Among them, optional, such as Fig. 9 As shown, the calculation module 802 includes an optimization module 8021 and an image processing module 8022 .

[0133] Optionally, in an implementation of this embodiment, when the acquisition module 801 acquires the feature point information of the image, it can be obtained through the image acquisition module 901 of the camera, such as Fig.10 shown.

[0134] Furthermore, the image acquisition module 901 is used to acquire multiple frames of images captured / scanned by the vehicle during driving, and send the multiple frames of images to the acquisition module 801 .

[0135] The image processing module 8022 is used to extract feature points from the multiple frames of images acquired by the acquisition module 801 to obtain at least one feature point of each frame of image; and perform feature point matching based on the at least one feature point of each frame of image to obtain a matching relationship between feature points of images between frames. For example, it includes performing relevant preprocessing on the image, detecting feature information such as points, lines, intervals, etc., and matching processing of previous and next key frames.

[0136] Wherein, the algorithm for extracting feature points includes: a deep learning method, or an ORB feature point extraction algorithm.

[0137] Optionally, the camera is an image sensor, such as a front camera sensor of a mobile terminal or a vehicle-mounted terminal.

[0138] In addition, the acquisition module 801 is also used to acquire the positioning information of the vehicle through the GPS acquisition module 904 of the GPS sensor. The GPS acquisition module 905 is used to acquire the positioning information of the vehicle and report it to the server.

[0139] Optionally, in another implementation of the present embodiment, the acquisition module 801 acquires the first speed information of the vehicle, specifically including: a speed acquisition module 902 in a wheel speed meter or wheel speed sensor, for acquiring the second speed information of the vehicle, the second speed information including the second linear speed and / or second angular velocity of the vehicle; an optimization module 903 optimizes the second speed information acquired by the speed acquisition module 902 to obtain the first speed information, and finally sends the first speed information to the acquisition module 801.

[0140] Alternatively, another implementation manner is that the optimization module 8021 in the calculation module 802 optimizes the second speed information to obtain the first speed information.

[0141] Furthermore, when optimizing the second speed information, the optimization module 8021 or the optimization module 903 in the wheel speed meter is also used to perform pre-integration processing on the second speed information to obtain the position and posture information of the vehicle; and process the second linear velocity and / or second angular velocity of the vehicle according to the position and posture information of the vehicle to obtain the first velocity information.

[0142] Optionally, in another implementation of the present embodiment, the calculation module 802 is also used to perform a first reprojection error calculation on the feature point information to obtain a second Jacobian matrix; update the second Jacobian matrix according to the first Jacobian matrix to obtain a third Jacobian matrix; and use the third Jacobian matrix to perform a second reprojection error calculation based on the feature points to obtain the first pose information.

[0143] Optionally, in another implementation of the present embodiment, the calculation module 802 is also used to determine whether the first reprojection error is within a preset threshold value based on the position and posture information of the vehicle; if so, the second reprojection error is calculated based on the feature points using the third Jacobian matrix to obtain the first posture information.

[0144] Optionally, in another implementation of the present embodiment, the calculation module 802 is also used to fit the first posture information calculated by the calculation module 802 and the positioning information of the vehicle reported by the GPS acquisition module 904 to obtain the first posture information in the GPS coordinate system and the GPS position information of the vehicle; perform posture graph optimization on the first posture information in the GPS coordinate system and the GPS position information of the vehicle to obtain the second posture information, and the second posture information is fitted with the GPS position information of the vehicle in the GPS coordinate system.

[0145] The composition module 803 is also used to optimize the optimized second posture information determined by the calculation module 802 and the GPS position information of the vehicle with geographic information, obtain the second posture information of the vehicle with precise scale and the point cloud set of each key frame, and finally form a 3D reconstructed point cloud map, thereby realizing global optimization within the region based on sparse point clouds and map construction of dense point clouds.

[0146] In addition, in the embodiment of the present device, Fig.10 The functions of the modules shown are similar to those described above. Figure 5 The method embodiment shown corresponds to, for example, the image acquisition module 901 in the camera is used to perform the aforementioned method step 110-1 and / or step 110-2, or the image processing module 8022 performs the aforementioned step 110-2. The speed acquisition module 902 in the wheel speed meter / wheel speed sensor can be used to implement the aforementioned step 110-3, and the optimization module 903 or the optimization module 8021 is used to implement the aforementioned step 110-4. The calculation module 802 and the composition module 803 can be used to implement the functions of all or part of the aforementioned steps 120 to 140.

[0147] Exemplary Hardware Devices

[0148] The hardware device of the embodiment of the present application is introduced below. The hardware device may be an electronic device, which may be a server or a mobile terminal, such as a vehicle computer, a vehicle-mounted device / vehicle-mounted terminal, etc.

[0149] When the hardware device is a server, such as Figure 3 As shown, a block diagram of a server according to an embodiment of the present application.

[0150] The processor 302 includes one or more processors or processing units, each of which can be a central processing unit (CPU) or other form of processing unit with data processing capability and / or instruction execution capability, and can control other components in the server, such as the communication interface 301, input and output devices 304, etc. to perform corresponding functions.

[0151] The memory 303 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc.

[0152] One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 302 may execute the program instructions to implement the point cloud map construction method of each embodiment of the present application described above and / or other desired functions. The computer-readable storage medium may also store sampling data reported by various sensors, such as image data, speed information, positioning information, etc.

[0153] In one example, components such as input / output device 304 and communication interface 301 may be interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0154] For example, at least one communication interface 301 on the server is connected to a transceiver for receiving various information reported by a vehicle computer or a mobile terminal device, including data and information collected by various sensors, and storing such information in a memory 303. The transceiver can also output various information to the outside through the communication interface 301, including the constructed point cloud map, so that the receiving end, the user equipment UE or the mobile terminal can receive and display the reconstructed 3D point cloud map.

[0155] In addition, the input / output device 304 may include, for example, a display, a speaker, a printer, and a communication network and remote output devices connected thereto.

[0156] Of course, to simplify, Figure 3 Only some of the components in the server related to the present application are shown, and components such as speakers, displays, transceivers, etc. are omitted. In addition, according to specific application conditions, the hardware device may also include any other appropriate components, which is not limited in this embodiment.

[0157] and, Figure 8 or Fig. 9Functions such as the acquisition module 801 , the calculation module 802 and the composition module 803 may be implemented by the processor 302 and / or the memory 303 , and the communication interface 301 .

[0158] In addition, if Figure 2 As shown, the vehicle computer 200 can also be used as an electronic device to realize all or part of the functions of the above-mentioned various sensors, such as the image sensor 204, the wheel speed sensor 205 and the GPS sensor 206. Fig.10 The functions of the image acquisition module 901, the speed acquisition module 902, the optimization module 903 and the GPS acquisition module 904 shown can be implemented by the above-mentioned sensors.

[0159] Exemplary computer program products and computer-readable storage media

[0160] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute some or all of the steps in the point cloud map construction method according to various embodiments of the present application described in the above "Exemplary Method" section of this specification.

[0161] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user computing device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0162] In addition, an embodiment of the present application may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes some or all of the steps in the point cloud map construction method according to various embodiments of the present application described in the above "Exemplary Method" section of this specification.

[0163] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0164] The basic principles of the present application are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present application are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present application. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, not for limitation, and the above details do not limit the present application to being implemented by adopting the above specific details.

[0165] The block diagrams of the devices, apparatuses, equipment, and systems involved in this application are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The word "such as" used here refers to the phrase "such as but not limited to", and can be used interchangeably with it.

[0166] It should also be noted that in the apparatus, device and method of the present application, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present application.

[0167] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present application. Therefore, the present application is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0168] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present application to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A method for constructing a point cloud map, the method comprising: Acquire feature point information of multiple frames of images, first speed information of the vehicle, and positioning information of the vehicle, wherein the feature point information includes: feature points of each frame of image and matching relationships between feature points between frames, and the first speed information of the vehicle includes a first linear speed and / or a first angular speed of the vehicle, and the first speed information is measured by a wheel speed sensor; Determining the first posture information of the vehicle according to the feature point information and the first speed information; Determining second position information of the vehicle according to the first position information and the positioning information of the vehicle; Reconstructing a point cloud using the feature point information of the multiple frames of images and the second pose information to obtain a point cloud map; Among them, determining the first posture information of the vehicle based on the feature point information and the first speed information includes: optimizing the feature point reprojection error based on the feature points of each frame image, the matching relationship between the feature points between frames, and the first speed information to obtain the first posture information.

2. The method according to claim 1, wherein: Acquiring first speed information of the vehicle, including: Acquiring second speed information of the vehicle reported by the wheel speed sensor, where the second speed information includes a second linear speed and / or a second angular speed of the vehicle; The second speed information is optimized to obtain the first speed information.

3. The method according to claim 2, wherein: Optimizing the second speed information to obtain the first speed information includes: Performing pre-integration processing on the second speed information to obtain position and posture information of the vehicle; The second linear velocity and / or second angular velocity of the vehicle is processed according to the position and posture information of the vehicle to obtain the first velocity information.

4. The method according to claim 3, wherein: The second linear velocity and / or second angular velocity of the vehicle is processed according to the position and posture information of the vehicle to obtain a first Jacobian matrix. Determining the first posture information of the vehicle according to the feature point information and the first speed information includes: Performing a first reprojection error calculation on the feature point information to obtain a second Jacobian matrix; Update the second Jacobian matrix according to the first Jacobian matrix to obtain a third Jacobian matrix; A second reprojection error calculation is performed using the third Jacobian matrix to obtain the first pose information.

5. The method according to claim 4, wherein: Using the third Jacobian matrix to perform a second reprojection error calculation to obtain the first pose information includes: When it is determined that the first reprojection error is within a preset threshold value according to the position and posture information of the vehicle, the second reprojection error is calculated using the third Jacobian matrix to obtain the first posture information.

6. The method according to claim 4, wherein: The positioning information includes GPS location information; Determining second posture information of the vehicle according to the first posture information and the positioning information of the vehicle includes: Fitting the first posture information and the GPS position information to obtain the first posture information and the GPS position information of the vehicle in a GPS coordinate system; Performing pose graph optimization on the first pose information in the GPS coordinate system and the GPS position information of the vehicle to obtain the second pose information, where the second pose information is fitted with the GPS position information of the vehicle in the GPS coordinate system.

7. The method according to any one of claims 1 to 6, wherein: Get feature point information of multiple frames of images, including: Extracting feature points from the multiple frames of images taken by the vehicle during driving to obtain at least one feature point for each frame of image; Feature point matching is performed based on at least one feature point of each frame of image to obtain a matching relationship between feature points of images between frames.

8. A point cloud map construction device, the device comprising: Acquisition module, calculation module and composition module, among which, The acquisition module is used to acquire feature point information of multiple frames of images, first speed information of the vehicle and positioning information of the vehicle, wherein the feature point information includes: feature points of each frame of image and matching relationship between feature points between frames, the first speed information of the vehicle includes a first linear speed and / or a first angular speed of the vehicle, and the first speed information is measured by a wheel speed sensor; The calculation module is used to determine the first posture information of the vehicle according to the feature point information and the first speed information obtained from the acquisition module, and determine the second posture information of the vehicle according to the first posture information and the positioning information of the vehicle obtained by the acquisition module; The composition module is used to reconstruct the point cloud using the feature point information of the multiple frames of images and the second posture information obtained by the calculation module to obtain a point cloud map; The calculation module is specifically used to optimize the feature point reprojection error according to the feature points of each frame image, the matching relationship between the feature points between frames and the first speed information to obtain the first posture information.

9. A computer-readable storage medium storing a computer program. When the computer program is executed, the point cloud map construction method described in any one of claims 1 to 7 is implemented.

10. An electronic device, comprising: processor and memory, The memory is used to store executable instructions; The processor is used to read the executable instructions from the memory and execute the instructions to implement the point cloud map construction method described in any one of claims 1 to 7 above.

Images