Map offline optimization method and electronic device

Abstract

The embodiment of the present disclosure discloses a map offline optimization method and an electronic device. The method comprises the following steps: determining visual point cloud points belonging to a set plane in a visual point cloud map; determining laser point cloud points belonging to the set plane in a laser point cloud map; performing an alignment operation on the visual point cloud points and the laser point cloud points to obtain a z-coordinate reference value of the visual point cloud points; and optimizing the visual point cloud map according to the z-coordinate reference value of the visual point cloud points. The present disclosure realizes the purpose of offline optimization of the visual point cloud map under the condition of no true value data, improves the accuracy of the visual point cloud map, and improves the overall accuracy and stability of the intelligent driving positioning system.

Description

Technical Field

[0001] This disclosure relates to the field of map construction technology, and in particular to a map offline optimization method and electronic device. Background Technology

[0002] In the field of intelligent driving, visual positioning is an important positioning method. Visual point cloud maps are a key component of visual positioning, and their quality directly determines the accuracy of visual positioning.

[0003] visually Figure 1 Visual maps are typically constructed using visual sensors. They are built through front-end feature extraction and back-end optimization of camera pose and 3D landmarks. Components such as GPS, IMU, and wheel speedometers can be used for assistance during the construction process. The accuracy of the constructed visual maps usually still has room for improvement. Further offline optimization will help improve the accuracy of the visual maps, thereby improving the accuracy of localization when using them.

[0004] Existing offline optimization methods for visual point cloud maps often compare the ground truth point cloud with the point cloud in the visual point cloud map to achieve map optimization. However, obtaining the ground truth point cloud is quite difficult in reality. Some methods use ground truth trajectories to constrain the trajectory of the visual point cloud map, thereby optimizing the quality of the visual point cloud map; similarly, obtaining ground truth trajectories consistent with those used during map construction is also quite difficult. Some methods use key points in the scene (such as corners or areas with obvious road signs) to correct the visual point cloud map; however, key points differ in different scenes, and some scenes may not contain usable high-quality key points.

[0005] Therefore, the method for offline optimization of visual point cloud maps needs further improvement. Summary of the Invention

[0006] To address the aforementioned technical problems, or at least partially address them, this disclosure provides a map offline optimization method and electronic device. This method optimizes visual point cloud maps offline in the absence of ground truth data, thereby improving the accuracy of visual point cloud maps and ultimately enhancing the overall accuracy and stability of the intelligent driving positioning system.

[0007] In a first aspect, embodiments of this disclosure provide a map offline optimization method, the method comprising:

[0008] Identify visual point cloud points in the visual point cloud map that belong to a defined plane;

[0009] Identify laser point cloud points in the laser point cloud map that belong to the defined plane;

[0010] Alignment operations are performed on the visual point cloud points and the laser point cloud points to obtain the z-coordinate reference value of the visual point cloud points;

[0011] The visual point cloud map is optimized based on the z-coordinate reference value of the visual point cloud points.

[0012] Secondly, embodiments of this disclosure also provide a map offline optimization device, the device comprising:

[0013] The first determining module is used to determine visual point cloud points in the visual point cloud map that belong to a set plane; the second determining module is used to determine laser point cloud points in the laser point cloud map that belong to the set plane; the alignment module is used to perform an alignment operation on the visual point cloud points and the laser point cloud points to obtain the z-coordinate reference value of the visual point cloud points; and the optimization module is used to optimize the visual point cloud map based on the z-coordinate reference value of the visual point cloud points.

[0014] Thirdly, embodiments of this disclosure also provide an electronic device, the electronic device comprising: one or more processors; a storage device for storing one or more programs; and when the one or more programs are executed by the one or more processors, causing the one or more processors to implement the map offline optimization method as described above.

[0015] Fourthly, embodiments of this disclosure also provide a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the map offline optimization method as described above.

[0016] This disclosure provides an offline map optimization method. The method involves identifying visual point cloud points belonging to a defined plane in a visual point cloud map; identifying laser point cloud points belonging to the defined plane in a laser point cloud map; aligning the visual point cloud points and the laser point cloud points to obtain a z-coordinate reference value for each visual point cloud point; and optimizing the visual point cloud map based on the z-coordinate reference values. This method achieves offline optimization of the visual point cloud map even without ground truth data, improving the accuracy of the visual point cloud map and consequently enhancing the overall accuracy and stability of the intelligent driving positioning system. Attached Figure Description

[0017] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0018] Figure 1 This is a comparative diagram of a visual point cloud map and a laser point cloud map in an embodiment of this disclosure;

[0019] Figure 2 This is a flowchart of an offline map optimization method according to an embodiment of this disclosure;

[0020] Figure 3 This is a schematic diagram illustrating the alignment of visual point cloud points with laser point cloud points in an embodiment of this disclosure;

[0021] Figure 4 This is a flowchart illustrating another offline map optimization method according to an embodiment of this disclosure;

[0022] Figure 5 This is a schematic diagram of the structure of an electronic device according to an embodiment of this disclosure. Detailed Implementation

[0023] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0024] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0025] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0026] Existing offline optimization methods for visual point cloud maps often compare ground truth point clouds with those in the visual point cloud map to achieve map optimization. However, obtaining ground truth point clouds is quite difficult in reality. On the other hand, in autonomous driving scenarios, laser point cloud maps, as part of laser localization, can generally be constructed together with the visual point cloud map in the same batch of offline data to ensure the stability of overall localization. Furthermore, the point clouds in laser point cloud maps are dense and highly accurate, such as... Figure 1As shown, the left image is a schematic diagram of a visual point cloud map, and the right image is a schematic diagram of a laser point cloud map. It is quite obvious that the point cloud in the laser point cloud map is denser. Furthermore, planes are relatively easy to identify features and are readily available in both images and laser point cloud maps. Based on these considerations, this invention proposes an offline optimization method for visual point cloud maps based on planar constraints. This method acquires and aligns the planar point clouds in the visual point cloud map and the laser point cloud map, using the planar point clouds from the laser point cloud map as constraints to correct the point clouds in the visual point cloud map, thereby optimizing the quality of the visual point cloud map.

[0027] In other words, the offline map optimization method provided in this disclosure is specifically an offline optimization method for visual point cloud maps. This method is executed after the visual point cloud map is constructed but before positioning or other further operations are performed using the constructed visual point cloud map, with the aim of further improving the accuracy of the visual point cloud map. The offline map optimization method provided in this disclosure achieves the goal of offline optimization of visual point cloud maps in the absence of ground truth data, improving the accuracy of the visual point cloud map and thus improving the overall accuracy and stability of the intelligent driving positioning system.

[0028] Figure 2 This is a flowchart of an offline map optimization method according to an embodiment of the present disclosure. This method can be executed by an offline map optimization device, which can be implemented in software and / or hardware, and can be configured in an electronic device. Figure 2 As shown, the method may specifically include the following steps:

[0029] S110. Determine the visual point cloud points in the visual point cloud map that belong to the specified plane.

[0030] Among these features, planes are relatively easy to identify and are readily available in both images and laser point cloud maps. Based on these considerations, this invention proposes an offline optimization method for visual point cloud maps based on plane constraints. For example, the designated plane can be the ground. Compared to other planes, the ground appears more frequently and has better stability in image frame sequences; therefore, the ground is selected as the reference reference in this embodiment.

[0031] A visual point cloud map can be a map constructed based on the pose of a visual sensor (such as a camera) and visual data (such as images) collected by the visual sensor. Understandably, components such as GPS (Global Positioning System), IMU (Inertial Measurement Unit), and wheel speedometers can be used to assist in the construction of a visual point cloud map.

[0032] In some embodiments, VSLAM (Visual Simultaneous Localization and Mapping) related methods (such as pure vision methods, or methods that fuse multiple sensors—such as cameras, IMUs, wheel speedometers, GPS, etc.) are used to construct visual point cloud maps. This disclosure does not limit the method used to construct the visual point cloud map. Regardless of the mapping method used, some calculation errors and system cumulative errors will exist during the mapping process; therefore, the accuracy of the constructed visual point cloud map still has room for improvement.

[0033] Specifically, when saving the visual point cloud map, an index structure is established for each 3D point cloud point, and the structure of the constituent elements of this index structure is shown in Table 1 below:

[0034] Table 1

[0035]

[0036] The purpose of establishing the above index structure is to facilitate the subsequent determination of visual point cloud points belonging to the defined plane in the visual point cloud map.

[0037] For example, step S110, determining the visual point cloud points belonging to the designated plane in the visual point cloud map, includes the following steps 111-113:

[0038] 111. Determine the planar pixels belonging to the set plane in each image acquired by the vehicle-mounted vision sensor.

[0039] Optionally, road surface pixels can be extracted from images using traditional computer vision methods or deep learning-based semantic segmentation methods. Extracting road surface pixels from images yields a set of 2D road surface pixels and a set of 2D non-road surface pixels for each image.

[0040] To facilitate statistical analysis, the pixels at each position (x, y) in the i-th image are binarized to generate the corresponding ground point mask image.

[0041]

[0042] That is, if the pixel at position (x, y) is a ground point, then If the pixel at position (x, y) is not a ground point, then

[0043] 112. When constructing the visual point cloud map, establish an association relationship between each point in the visual point cloud map and the image.

[0044] The specific form of the association can be an index structure as shown in Table 1 above.

[0045] 113. Determine the visual point cloud points based on the planar pixels and the associated relationships.

[0046] Furthermore, step 113, determining the visual point cloud points based on the planar pixels and the association relationship, includes the following steps 1131-1133:

[0047] 1131. For the current point cloud point in the visual point cloud map, determine multiple target images including target pixels based on the association relationship, wherein the target pixels are the pixels corresponding to the current point cloud point in the image.

[0048] 1132. Traverse each of the target images to determine the first number of times the target pixel in each target image is a pixel in the planar pixels and the second number of times it is not a pixel in the planar pixels.

[0049] 1133. If the first count is greater than the second count, then the current point cloud point is determined to be the visual point cloud point; or, if the first count reaches a threshold, then the current point cloud point is determined to be the visual point cloud point, wherein the threshold is determined based on the sum of the first count and the second count.

[0050] In one specific implementation, the road surface 2D pixel mask image generated in step 111 above is used. For each 3D point in the visual point cloud map, a voting counter is defined to count the number of votes for whether the current 3D point is a ground point, and the PixelsSet in Table 1 above is used for filtering. Specifically, each element in the PixelsSet is traversed, and the corresponding mask image from step 111 is found based on the image ID. If pixel coordinates P 2d corresponding If the count is 1, the current 3D point's vote counter is incremented by 1; otherwise, it is decremented by 1. Finally, the vote counter count for each 3D point is tallied. If the count is positive, the corresponding 3D point is added to the road surface 3D point set. In other words, if the count is positive, the corresponding 3D point is determined to be a visual point cloud point, ultimately forming the road surface 3D point set (i.e., the visual point cloud point) of the visual point cloud map. Note: A voting counter is used because a 3D spatial point typically appears in multiple frames of images. In step 111, different image planes may yield different extraction results for the same spatial point; for example, some... It could be 1, or it could be -1. Therefore, the same 3D point in the visual point cloud map is accumulated to eliminate the error in the extraction of road surface pixels in the image space to a certain extent, improve the extraction accuracy, and thus ensure the final map optimization effect.

[0051] S120. Determine the laser point cloud points in the laser point cloud map that belong to the set plane.

[0052] The laser point cloud map can be a map constructed based on the pose of a lidar and the point cloud data it collects. This invention selects laser point cloud data acquired synchronously with images and uses LSLAM (Lidar Simultaneous Localization and Mapping) related methods (such as pure laser methods, or methods that fuse multiple sensors (such as IMU, wheel speedometer, GPS, etc.)) to construct the laser point cloud map.

[0053] For example, methods such as height threshold filtering, point cloud segmentation, and normal vector filtering can be used to extract laser point cloud points belonging to the defined plane. Taking the defined plane as the ground as an example, the laser point cloud points belonging to the defined plane form a road surface point cloud, ultimately obtaining a 3D point set of the road surface in the laser point cloud map.

[0054] S130. Align the visual point cloud points and the laser point cloud points to obtain the z-coordinate reference value of the visual point cloud points.

[0055] For example, step S130 can be implemented through the following steps 131-133:

[0056] 131. Obtain the visual pose of the visual sensor and the laser pose of the lidar aligned at the same timestamp.

[0057] 132. For each aligned visual pose and laser pose at each timestamp, construct a first cube centered on the position coordinates (x, y, z) in the visual pose, and construct a second cube centered on the position coordinates (x, y, z) in the laser pose. The size of the first cube and the second cube are the same. The length of the first cube is determined based on the width of the road surface. The width of the first cube is determined based on the magnitude of the visual direction vector of the visual sensor. The direction of the width of the first cube is the trajectory direction of the visual pose, and the direction of the width of the second cube is the trajectory direction of the laser pose.

[0058] 133. Align the visual point cloud points and the laser point cloud points according to the first cube and the second cube to obtain the z-coordinate reference value of the visual point cloud points.

[0059] Furthermore, the step of aligning the visual point cloud points and the laser point cloud points based on the first cube and the second cube to obtain the z-coordinate reference value of the visual point cloud points includes:

[0060] Align the first cube with the second cube to obtain the transformation matrix when the second cube is transformed to a position that coincides with the first cube;

[0061] The laser point cloud points in the second cube space are transformed into the space of the first cube using the transformation matrix to obtain the transformed laser point cloud points.

[0062] For the current visual point cloud point, a target laser point cloud point that is closest to the current visual point cloud point is determined from the transformed laser point cloud points in the xy plane, and the z-coordinate value of the target laser point cloud point is determined as the z-coordinate reference value of the current visual point cloud point. The current visual point cloud point is any one of the visual point cloud points in the first cube space.

[0063] In one specific embodiment, step S130 above can be implemented through the following process:

[0064] Pose alignment: First, traverse the trajectory TR of the visual point cloud map. v (This trajectory is generated and saved synchronously when constructing the visual point cloud map), for the current visual pose. Based on the timestamp in the trajectory TR on the laser point cloud map l (This trajectory is generated and saved synchronously when constructing the laser point cloud map) The search term is for the laser pose that is most recent in time. If the most recent laser pose is similar to the current visual pose... If the time difference is greater than the first threshold (e.g., 30ms), the corresponding laser point cloud pose is calculated using linear interpolation. (i.e., the laser pose aligned with the current visual pose at the same timestamp). Visual poses of the visual sensors aligned at the same timestamp. Laser pose with lidar The expression can be in the following forms:

[0065]

[0066]

[0067] Among them, the translation part that records the visual pose is: This value also represents the coordinates (or position coordinates) of a point in an absolute coordinate system.

[0068] Construct two cubic spaces: Then, using the translation portion of the pose as the center (i.e., the position coordinates in the pose), construct two cubes of the same size. The length of the cube is a second threshold multiple of the road width (e.g., 1.5 times the road width). The height of the cube is a third threshold (e.g., 1m). The width of the cube follows its respective trajectory direction. Let the translation vector of the current frame be denoted as... The translation vector for the next frame is The direction of the cube's width is a vector. The direction and cube width are both taken from the visual direction vector. The fourth threshold (e.g., 1.5) of the modulus length. Based on the length, width, and height of the cube above, two cubic spaces can be determined, denoted as C. v and C l .

[0069] Aligning the two cubes: Because the visual trajectory and the laser trajectory are different, although the two cubes are the same size, their spatial positions are slightly different. Here, we spatially align the two cubes (using existing alignment methods), resulting in C... l Transform to C v The transformation matrix when they coincide is denoted as T. cube .

[0070] Ground point cloud alignment: based on two cubic spaces C v and C l The location is selected from the visual ground 3D point cloud and the laser ground 3D point cloud, respectively, and the point in the corresponding cube space is selected. The C corresponding to the current frame of the visual ground 3D point cloud is... v The point in the middle is marked as C corresponding to the current frame of the laser ground 3D point cloud v The point in the middle is marked as Use T cube laser point cloud The transformation is denoted as:

[0071]

[0072] Finding the reference value of the z-axis for points in a visual point cloud map within a laser point cloud map: for a set of points and Let the midpoints of the two sets be represented as follows:

[0073]

[0074]

[0075] Traversal Every point, in the xy plane, in In a set, find the point that is closest to you, denoted as . Use its z-axis coordinate value As P v Using the reference value of the z-axis, the prior coordinates of the current ground point are constructed, denoted as:

[0076]

[0077] Note: For example Figure 1 As shown, the left side is a visual point cloud map, and the right side is a laser point cloud map. Generally, the laser point cloud map is much denser than the visual point cloud map. Therefore, most ground points in the visual point cloud map can find reference values ​​in the laser point cloud map. For example, see... Figure 3 The diagram shows an alignment of visual point cloud points with laser point cloud points.

[0078] S140. Optimize the visual point cloud map based on the z-coordinate reference value of the visual point cloud points.

[0079] For example, optimizing the visual point cloud map based on the z-coordinate reference value of the visual point cloud points includes:

[0080] The visual reprojection error is determined using the z-coordinate reference values ​​of the visual point cloud points. A cost function is constructed based on at least the visual reprojection error using a sliding window approach, and the cost function is solved using the least squares method to obtain optimized map points. These optimized map points form an optimized visual point cloud map.

[0081] The step of determining the visual reprojection error using the z-coordinate reference value of the visual point cloud points includes:

[0082] The original z-coordinate value of the visual point cloud point is replaced with the z-coordinate reference value to obtain the coordinates of the replaced visual point cloud point.

[0083] The visual point cloud points are reprojected using the coordinates of the replaced visual point cloud points to obtain the reference coordinates of the corresponding pixel points in the corresponding image.

[0084] The difference between the original coordinates of the visual point cloud points and the reference coordinates of the corresponding pixels in the corresponding image is determined as the visual reprojection error.

[0085] The method of using a sliding window to construct a cost function based on the visual reprojection error includes:

[0086] The cost function is constructed according to the following expression:

[0087]

[0088] Where f(x) represents the cost function, a sliding window includes the visual poses of m visual sensors, the 1st to n1st point cloud points in the sliding window are the visual point cloud points, and the n1+1st to nth point cloud points in the sliding window are point cloud points that do not belong to the set plane in the visual point cloud map.

[0089] For the i-th visual pose ξ i The j-th point cloud point P j If P j For points that do not belong to the defined plane, their coordinate values ​​are taken from the corresponding values ​​in the visual point cloud map. The corresponding visual reprojection error is If P j For points belonging to the defined plane, their coordinate values ​​are taken as follows: The x and y coordinates are the corresponding values ​​in the visual point cloud map, the z coordinate is the z coordinate reference value, and the corresponding visual reprojection error is... h(.) is the reprojection function, which calculates the pixel coordinates of the point cloud points in space onto the corresponding image by combining them with the visual pose of the vision sensor. Represents point cloud points The pixel coordinates projected onto the corresponding image. Represents point cloud points The pixel coordinates projected onto the corresponding image, z ij This represents the original coordinates of the corresponding pixel in the corresponding image for the corresponding point cloud point.

[0090] Specifically, after the above steps, the coordinates of the 3D ground points in the visual point cloud map have reference coordinates. This reference coordinate is used as the prior value for the 3D points of the road surface in the visual point cloud map. Next, a sliding window approach is used to optimize the visual point cloud map points and their corresponding poses, combining the aforementioned prior value and the visual reprojection error.

[0091] For the i-th camera pose ξ i The j-th map point (or landmark) P j If P j If a point is not a road surface point, its value is taken from the corresponding value in the original visual point cloud map. Reprojection error e ij It can be represented as:

[0092]

[0093] If P jIf it belongs to a road surface point, then its value is taken as the prior value of the road surface point. Reprojection error e ij It can be represented as:

[0094]

[0095] Among them, z ij Let be the 2D coordinates of the j-th 3D landmark in the i-th image plane. Assume a sliding window has m camera poses and n visual 3D landmarks, where road surface landmarks are from the 1st to the n1th, and non-road surface landmarks are from the n1+1th to the nth. All variables to be optimized are represented as follows:

[0096]

[0097] The overall cost function is the sum of the reprojection errors of all points. A least squares problem is then constructed, specifically represented as:

[0098]

[0099] Then, by traversing all images and solving the least squares problem, the optimal values ​​of all variables to be optimized can be obtained as follows:

[0100]

[0101] This yields the optimized visual point cloud map.

[0102] Furthermore, when constructing the cost function, in addition to the visual reprojection error, IMU error can also be incorporated. That is, constructing the cost function based at least on the visual reprojection error includes constructing a cost function based on both the visual reprojection error and the inertial measurement unit (IMU) error. The more error forms incorporated, the higher the accuracy of the final optimized map points.

[0103] The offline map optimization method provided in this disclosure utilizes the high precision and density of laser point cloud maps to perform offline optimization of visual point cloud maps in the absence of ground truth data, thereby improving the overall accuracy and stability of the intelligent driving positioning system.

[0104] Based on the above embodiments, refer to the following: Figure 4 The diagram illustrates another offline map optimization method, which specifically includes: VSLAM mapping based on images and related data, and extraction of road surface pixels in image space; obtaining a visual point cloud map through VSLAM mapping; obtaining 2D road surface pixels in image space through the extraction of road surface pixels; and combining the 2D road surface pixels in image space with the visual point cloud map to optimize the visual point cloud map. Figure 33D road surface points are filtered to obtain a set of 3D road surface points in the visual point cloud map. Simultaneously, a laser point cloud map is obtained through SLAM mapping based on the laser point cloud and related data, and 3D road surface points are extracted from the laser point cloud map to obtain a set of 3D road surface points in the laser point cloud map. Next, the 3D road surface points in the visual point cloud map are aligned with those in the laser point cloud map to obtain reference z-coordinate values ​​for the 3D road surface points in the visual point cloud map. Based on these reference z-coordinate values, the visual point cloud map is optimized to obtain a high-quality visual point cloud map.

[0105] This disclosure also provides an offline map optimization device, which includes: a first determining module for determining visual point cloud points belonging to a set plane in a visual point cloud map; a second determining module for determining laser point cloud points belonging to the set plane in a laser point cloud map; an alignment module for performing an alignment operation on the visual point cloud points and the laser point cloud points to obtain a z-coordinate reference value of the visual point cloud points; and an optimization module for optimizing the visual point cloud map based on the z-coordinate reference value of the visual point cloud points.

[0106] The map offline optimization apparatus provided in this disclosure embodiment can execute the steps in the map offline optimization method provided in this disclosure method embodiment, and can obtain the same beneficial effects, which will not be repeated here.

[0107] Figure 5 This is a schematic diagram of the structure of an electronic device according to an embodiment of this disclosure. See below for details. Figure 5 It shows a schematic diagram of a structure suitable for implementing the electronic device 500 in the embodiments of this disclosure. Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0108] like Figure 5 As shown, the electronic device 500 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 501, which can perform various appropriate actions and processes to implement the methods of the embodiments described herein, based on a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of the electronic device 500. The processing device 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0109] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts, thereby implementing the offline map optimization method as described above. In such embodiments, the computer program can be downloaded and installed from a network via communication device 509, or installed from storage device 508, or installed from ROM 502. When the computer program is executed by processing device 501, it performs the functions defined in the methods of embodiments of this disclosure.

[0110] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0111] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: acquire perception results output by a 3D perception algorithm that correspond one-to-one with the frame of data to be perceived, the perception results including the tracking numbers of targets perceived based on the 3D perception algorithm; arrange the perception results of targets with the same tracking numbers in chronological order to obtain a target tracking list; determine a quality assessment index based on the target tracking list to characterize the stability of the perception results; and assess the quality of the perception results according to the quality assessment index.

[0112] Optionally, when one or more of the above-described procedures are executed by the electronic device, the electronic device may also perform other steps described in the above embodiments.

[0113] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0114] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

Claims

1. A method for offline map optimization, characterized in that, The method includes: Identify visual point cloud points in the visual point cloud map that belong to a defined plane; Identify laser point cloud points in the laser point cloud map that belong to the defined plane; Alignment operations are performed on the visual point cloud points and the laser point cloud points to obtain the z-coordinate reference value of the visual point cloud points; The original z-coordinate value of the visual point cloud point is replaced with the z-coordinate reference value to obtain the coordinates of the replaced visual point cloud point. The visual point cloud points are reprojected using the coordinates of the replaced visual point cloud points to obtain the reference coordinates of the corresponding pixel points in the corresponding image. The difference between the original coordinates of the visual point cloud points and the reference coordinates of the corresponding pixels in the corresponding image is determined as the visual reprojection error. Using a sliding window approach, a cost function is constructed based at least on the visual reprojection error, and the cost function is solved using the least squares method to obtain optimized map points.

2. The method according to claim 1, characterized in that, The determination of visual point cloud points belonging to a defined plane in the visual point cloud map includes: Identify the planar pixels belonging to the defined plane in each image acquired by the vehicle-mounted vision sensor; When constructing the visual point cloud map, an association relationship is established between each point in the visual point cloud map and the image. The visual point cloud points are determined based on the planar pixels and the associated relationships.

3. The method according to claim 2, characterized in that, Determining the visual point cloud points based on the planar pixels and the correlation relationships includes: For the current point cloud point in the visual point cloud map, multiple target images including target pixels are determined according to the association relationship, where the target pixels are the pixels corresponding to the current point cloud point in the image; Each target image is traversed to determine a first number of times that the target pixel in each target image is a pixel in the planar pixels and a second number of times that it is not a pixel in the planar pixels; If the first count is greater than the second count, then the current point cloud point is determined to be the visual point cloud point; or, if the first count reaches a threshold, then the current point cloud point is determined to be the visual point cloud point. The threshold is determined based on the sum of the first count and the second count.

4. The method according to claim 1, characterized in that, The step of aligning the visual point cloud points and the laser point cloud points to obtain the z-coordinate reference value of the visual point cloud points includes: Obtain the visual pose of the visual sensor and the laser pose of the LiDAR at the same timestamp: For each time-stamp aligned visual pose and laser pose, a first cube is constructed with the position coordinates of the visual pose in the absolute coordinate system as the center, and a second cube is constructed with the position coordinates of the laser pose in the absolute coordinate system as the center. The size of the first cube and the second cube are the same. The length of the first cube is determined based on the width of the road surface. The width of the first cube is determined based on the magnitude of the visual direction vector of the visual sensor. The direction of the width of the first cube is the trajectory direction of the visual pose, and the direction of the width of the second cube is the trajectory direction of the laser pose. Alignment operations are performed on the visual point cloud points and the laser point cloud points based on the first cube and the second cube to obtain the z-coordinate reference value of the visual point cloud points.

5. The method according to claim 4, characterized in that, The step of aligning the visual point cloud points and the laser point cloud points according to the first cube and the second cube to obtain the z-coordinate reference value of the visual point cloud points includes: Align the first cube with the second cube to obtain the transformation matrix when the second cube is transformed to a position that coincides with the first cube; The laser point cloud points in the second cube space are transformed into the space of the first cube using the transformation matrix to obtain the transformed laser point cloud points. For the current visual point cloud point, a target laser point cloud point that is closest to the current visual point cloud point is determined from the transformed laser point cloud points in the xy plane, and the z-coordinate value of the target laser point cloud point is determined as the z-coordinate reference value of the current visual point cloud point. The current visual point cloud point is any one of the visual point cloud points in the first cube space.

6. The method according to claim 1, characterized in that, The method of using a sliding window to construct a cost function based at least on the visual reprojection error includes: The cost function is constructed according to the following expression: Where f(x) represents the cost function, a sliding window includes the visual poses of m visual sensors, the first to n1th point cloud points in the sliding window are the visual point cloud points, and the (n1+1)th to nth point cloud points in the sliding window are point cloud points that do not belong to the set plane in the visual point cloud map. For the i-th visual pose The j-th point cloud point ,like For points that do not belong to the defined plane, their coordinate values ​​are taken from the corresponding values ​​in the visual point cloud map. The corresponding visual reprojection error is : ;like For points belonging to the defined plane, their coordinate values ​​are taken as follows: , The x and y coordinates are the corresponding values ​​in the visual point cloud map, the z coordinate is the z coordinate reference value, and the corresponding visual reprojection error is... : h(.) is the reprojection function, used to calculate the pixel coordinates of the point cloud points in space onto the corresponding image by calculating the visual pose of the vision sensor. Represents point cloud points The pixel coordinates projected onto the corresponding image. Represents point cloud points The pixel coordinates projected onto the corresponding image. This represents the original coordinates of the corresponding pixel in the corresponding image for the corresponding point cloud point.

7. The method according to claim 1, characterized in that, The construction of the cost function based at least on the visual reprojection error includes: A cost function is constructed based on the visual reprojection error and the inertial measurement unit (IMU) error.

8. An electronic device, characterized in that, The electronic device includes: One or more processors; Storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-7.

Images