A point cloud processing method, electronic device, and program product

Abstract

The present specification provides a point cloud processing method, an electronic device and a program product, acquiring a point cloud collected by a point cloud collection device and an image collected by an image collection device, wherein the collection range of the image matches the collection range of the point cloud. The first pixel position of the target object in the image is obtained, and according to the relative position relationship between the two collection devices, the target point cloud matched with the first pixel position is determined from the point cloud as the point cloud of the target object, thereby completing the process of labeling the target object in the point cloud. Since the image collected by the image collection device is more intuitive than the point cloud data and has higher recognizability, the image with the labeled target object is used to assist the point cloud in labeling the target object, which can effectively reduce the difficulty of direct labeling in the point cloud and improve the labeling accuracy and the accuracy of the high-precision map constructed.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of data processing technology, and in particular to a point cloud processing method, electronic device, and program product. Background Technology

[0002] Autonomous driving relies on high-precision maps for positioning and navigation. Building a high-precision map first requires map-collecting vehicles to gather data on various road features, including image data and point cloud data. The collected data is then transformed into usable map data to construct the high-precision map. During this process, various road features need to be labeled from the collected point cloud data; however, point cloud labeling is challenging, reducing its accuracy and consequently lowering the accuracy of the constructed high-precision map. Summary of the Invention

[0003] This specification provides a point cloud processing method, electronic device, and program product to reduce the difficulty of point cloud annotation.

[0004] According to a first aspect of the embodiments of this specification, a point cloud processing method is provided, the method comprising:

[0005] The method acquires point clouds collected by a point cloud acquisition device and images collected by an image acquisition device; wherein the acquisition range of the image matches the acquisition range of the point cloud.

[0006] Obtain the first pixel position of the target object in the image;

[0007] Based at least on the relative positional relationship between the point cloud acquisition device and the image acquisition device, the target point cloud of the target object that matches the position of the first pixel is determined from the point cloud.

[0008] In some examples, determining the target point cloud of the target object matching the first pixel position from the point cloud, at least based on the relative positional relationship between the point cloud acquisition device and the image acquisition device, includes:

[0009] Obtain the second pixel position of the target object in the projected image; wherein, the projected image is the image obtained by projecting the two-dimensional point cloud onto the ground after identifying the two-dimensional point cloud from the point cloud;

[0010] Based on the relative positional relationship, a first point cloud that matches the position of the first pixel is determined from the two-dimensional point cloud;

[0011] Based on the coordinate transformation relationship between the two-dimensional point cloud and the projected image, a second point cloud matching the position of the second pixel is determined from the two-dimensional point cloud;

[0012] The target point cloud is determined based on the first point cloud and the second point cloud.

[0013] In some examples, the points in the point cloud carry the three-dimensional position information of the point; the three-dimensional information includes the elevation information of the point.

[0014] After identifying the two-dimensional point cloud from the point cloud, projecting the two-dimensional point cloud onto the ground includes:

[0015] The point cloud whose elevation information is less than a preset threshold is identified as the two-dimensional point cloud.

[0016] The projected image is generated based on the two-dimensional position information of the two-dimensional point cloud; wherein the two-dimensional position information is obtained based on the three-dimensional position information of the two-dimensional point cloud.

[0017] In some examples, determining the target point cloud based on the first point cloud and the second point cloud includes:

[0018] The first point cloud and the second point cloud are fused to obtain the target point cloud.

[0019] In some examples, determining the target point cloud of the target object matching the first pixel position from the point cloud, at least based on the relative positional relationship between the point cloud acquisition device and the image acquisition device, includes:

[0020] Based on the relative positional relationship and the size of the image, a third point cloud is segmented from a frame of point cloud acquired by the point cloud acquisition device;

[0021] Based on the relative positional relationship, the target point cloud of the target object that matches the position of the first pixel is determined from the third point cloud.

[0022] In some examples, obtaining the first pixel position of the target object in the image includes:

[0023] Perform semantic segmentation on the image to obtain the first pixel position of the target object in the image; and / or

[0024] Obtain the first pixel position stored in the image after the target object is manually marked.

[0025] In some examples, determining the target point cloud of the target object matching the first pixel position from the point cloud, at least based on the relative positional relationship between the point cloud acquisition device and the image acquisition device, includes:

[0026] Based on the first pixel position, a mask image corresponding to the image is generated; the mask image is marked with the first pixel position and is used to remove other image content except for the target object;

[0027] Based on the relative positional relationship between the point cloud acquisition device and the image acquisition device, the point cloud is projected onto the mask image;

[0028] The target point cloud of the target object that matches the position of the first pixel is determined from the point cloud.

[0029] In some examples, determining the target point cloud of the target object matching the first pixel position from the point cloud includes:

[0030] Acquire multiple frames of point clouds collected within a preset time period, and determine the point cloud that matches the position of the first pixel from each frame of the point cloud;

[0031] The point clouds matched in each frame are fused together to obtain the target point cloud of the target object.

[0032] In some examples, the points in the point cloud carry relative position information of the points with respect to the point cloud acquisition device; the method further includes:

[0033] Based on the geographic location information recorded by the point cloud acquisition device when acquiring the point cloud, and the relative position information of the target point cloud, the geographic location information of the target object is obtained;

[0034] A map is constructed based on the geographic location information of the target object.

[0035] In some examples, constructing a map based on the geographic location information of the target object includes:

[0036] Based on the type of the target object, a visualization model corresponding to the target object is determined from a visualization model library; wherein, the visualization model library stores visualization models corresponding to different types of objects; a map is constructed based on the geographic location information of the target object and the visualization model; and / or

[0037] Based on the target point cloud, obtain the shape information of the target object; construct a map based on the geographical location information of the target object and the shape information.

[0038] According to a second aspect of the embodiments of this specification, an electronic device is provided, the electronic device comprising:

[0039] processor;

[0040] Memory used to store processor-executable instructions;

[0041] Wherein, when the processor invokes the executable instructions, it implements the operation of any of the methods described in the first aspect above.

[0042] According to a third aspect of the embodiments of this specification, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of any of the methods described in the first aspect.

[0043] The technical solutions provided by the embodiments of this specification may include the following beneficial effects:

[0044] This specification provides a point cloud processing method, electronic device, and program product. It acquires a point cloud collected by a point cloud acquisition device and an image acquired by an image acquisition device, wherein the acquisition range of the image matches the acquisition range of the point cloud. The method obtains the first pixel position of a target object in the image, and based on the relative positional relationship between the two acquisition devices, determines the target point cloud that matches the first pixel position as the point cloud of the target object, thereby completing the process of marking the target object in the point cloud. Since the image acquired by the image acquisition device is more vivid and has higher recognizability than point cloud data, using the image with the marked target object to assist the point cloud in marking the target object can effectively reduce the difficulty of directly marking in the point cloud, improve the accuracy of the marking, and increase the accuracy of the constructed high-precision map.

[0045] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit the embodiments of this specification. Attached Figure Description

[0046] The accompanying drawings, which are incorporated in and form part of the embodiments of this specification, illustrate embodiments consistent with those of this specification and, together with the specification, serve to explain the principles of the embodiments of this specification.

[0047] Figure 1 This is a flowchart illustrating a point cloud processing method according to an embodiment of this specification.

[0048] Figure 2A This is a schematic diagram of the target object shown in this specification according to one embodiment.

[0049] Figure 2B This is a schematic diagram of the target object shown in this specification according to another embodiment.

[0050] Figure 2C This is a schematic diagram of the target object shown in this specification according to another embodiment.

[0051] Figure 3 This is a flowchart illustrating a point cloud processing method according to another embodiment of this specification.

[0052] Figure 4 This is a flowchart illustrating a point cloud processing method according to another embodiment of this specification.

[0053] Figure 5 This is a flowchart illustrating a point cloud processing method according to another embodiment of this specification.

[0054] Figure 6 This is a flowchart illustrating a point cloud processing method according to another embodiment of this specification.

[0055] Figure 7 This is a flowchart illustrating a point cloud processing method according to another embodiment of this specification.

[0056] Figure 8 This is a flowchart illustrating a point cloud processing method according to another embodiment of this specification.

[0057] Figure 9 This is a flowchart illustrating a point cloud processing method according to another embodiment of this specification.

[0058] Figure 10 This is a hardware structure diagram of an electronic device illustrated in this specification according to one embodiment. Detailed Implementation

[0059] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those described in this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the embodiments described in this specification as detailed in the appended claims.

[0060] The terminology used in the embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the embodiments of this specification. The singular forms “a,” “described,” and “the” as used in the embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0061] It should be understood that although the terms first, second, third, etc., may be used to describe various information in the embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first information may also be referred to as second information without departing from the scope of the embodiments of this specification, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0062] Autonomous driving relies on high-precision maps for positioning and navigation. Building a high-precision map first requires map-collecting vehicles to gather data on various road features. These features include ground features such as lane lines, stop lines, yield lines, ground arrows, text, and guide lines; and non-ground features such as poles, signs, traffic lights, traffic signs, gantries, and buildings—features at a certain distance from the ground. Map-collecting vehicles are specialized data acquisition devices for high-precision maps, typically equipped with LiDAR, image acquisition devices, Global Navigation Satellite System (GNSS), and Inertial Measurement Unit (IMU) data acquisition equipment. They can collect image data and point cloud data of various road features. The collected data is then converted into usable map data to construct a high-precision map. During this process, various road features need to be labeled from the collected point cloud data. However, point cloud labeling is difficult, reducing the accuracy of the labeling and consequently the accuracy of the constructed high-precision map.

[0063] Therefore, embodiments of this specification provide a point cloud processing method, including as follows: Figure 1 The steps shown are as follows:

[0064] Step 110: Acquire the point cloud acquired by the point cloud acquisition device and the image acquired by the image acquisition device;

[0065] The acquisition range of the image is matched with the acquisition range of the point cloud;

[0066] Step 120: Obtain the first pixel position of the target object in the image;

[0067] Step 130: Based at least on the relative positional relationship between the point cloud acquisition device and the image acquisition device, determine the target point cloud of the target object that matches the position of the first pixel from the point cloud.

[0068] Point cloud acquisition devices can include laser scanning equipment, such as LiDAR (Light Detection and Ranging), and of course, other similar devices with scanning capabilities. Image acquisition devices can include RGB cameras, infrared cameras, Time-of-Flight (ToF) cameras, and other similar devices with image acquisition capabilities.

[0069] The point cloud acquisition device and the image acquisition device can be mounted on the same mobile platform. The mobile platform can include, but is not limited to, mobile devices such as vehicles, aircraft, ships, unmanned aerial vehicles, and robots, such as the map acquisition vehicle mentioned above.

[0070] The point cloud processing method provided in the embodiments of this specification can be executed by a processor mounted on the aforementioned mobile platform, or by a computing node of a server cluster or distributed system.

[0071] The system acquires point clouds from a point cloud acquisition device and images from an image acquisition device, wherein the point cloud acquisition device and the image acquisition device can acquire point cloud and image data simultaneously, respectively. The acquisition range of the image matches the acquisition range of the point cloud. Acquisition range matching includes at least partial overlap between the acquisition ranges of the image and the point cloud. That is, the acquisition ranges of the image and the point cloud overlap (hereinafter referred to as overlapping range). Correspondingly, the field of view of the image acquisition device and the point cloud acquisition device at least partially overlap, that is, the field of view of the image acquisition device and the point cloud acquisition device overlap. For example, the acquisition range of the point cloud is larger than the acquisition range of the image and includes part or all of the acquisition range of the image. For example, the acquisition range of the image is larger than the acquisition range of the point cloud and includes part or all of the acquisition range of the point cloud. For another example, the acquisition range of the point cloud is equal to the acquisition range of the image, and the two acquisition ranges completely or partially overlap. In this way, objects within the overlapping range will be simultaneously acquired by the point cloud acquisition device and the image acquisition device, respectively, with image data and point cloud data being acquired.

[0072] After acquiring point cloud and image data collected simultaneously, the first pixel position of the target object in the image is obtained. The target object can be any object within the overlapping area. In the scenario of constructing a high-precision map, the target object can be any of the aforementioned road features. The process of annotating the target object in the image will be discussed below and will not be elaborated upon here. The first pixel position is used to characterize the location of the target object in the image. The first pixel position can include one or more pixel positions. The pixel position can correspond to the feature position of the target object.

[0073] like Figure 2AAs shown, if target objects belonging to a certain type have fixed shapes and sizes, such as lane lines, their positions in the image can be described using a single pixel location. This pixel location can correspond to other feature locations such as the center or apex of the target object. For example, Figure 2A The four lane lines 210 shown each have the same and fixed shape and size. Therefore, the position of a lane line 210 in the image can be represented by the pixel position of its center point 211. That is, the pixel position of the center point 211 is the first pixel position of the lane line 210.

[0074] like Figure 2B As shown, if target objects belonging to the same type have regular shapes but different sizes, such as traffic signs, their positions in the image can be described using multiple pixel locations. Each pixel location can correspond to a feature location such as the apex corner of the target object's shape. For example, Figure 2B The rectangular traffic sign 220 shown can be positioned in the image using the pixel positions of its four vertices 221-224. That is, the first pixel position of the traffic sign 220 includes the four pixel positions of its vertices 221-224.

[0075] like Figure 2C As shown, if the target object has an irregular shape, such as a gantry or building, its position in the image can be described using multiple pixel locations. For example, the pixel locations corresponding to the outline of the target object can be used to describe its position in the image. Alternatively, a regularly shaped bounding box can be used to mark the target object in the image, and then the pixel locations of the bounding box can be used to describe the target object's position in the image. For more information on how to describe the position of regularly shaped bounding boxes in an image, please refer to the above text. Figures 2A-2B The embodiments described herein will not be repeated here. For example, Figure 2C The gantry crane 230 shown can be positioned in an image using the pixel positions of its outline. That is, the first pixel position of the gantry crane 230 includes the pixel positions of its outline. Alternatively, the gantry crane 230 can be marked in the image using a rectangular bounding box, and then its position can be represented by the pixel positions of the four vertices of the rectangular bounding box. That is, the first pixel position of the gantry crane 230 includes the pixel positions of the four vertices of its rectangular bounding box.

[0076] After acquiring the first pixel position, based on the relative positional relationship between the point cloud acquisition device and the image acquisition device, the target point cloud of the target object matching the first pixel position can be determined from the point cloud. That is, the target point cloud determined from the point cloud that matches the first pixel position is the point cloud of the target object, thus completing the process of marking the target object in the point cloud. The relative positional relationship between the point cloud acquisition device and the image acquisition device can be pre-defined and stored. In some embodiments, when determining the target point cloud matching the first pixel position, at least the pixel set corresponding to the target object in the image can be determined based on the first pixel position. Then, the point cloud corresponding to the pixel set is determined from the point cloud as the target point cloud, i.e., the point cloud of the target object. It is understood that the pixel set includes multiple pixels; therefore, determining the point cloud corresponding to the pixel set as the target point cloud involves determining the point corresponding to each pixel in the pixel set one by one, and the set of determined points is the target point cloud.

[0077] As mentioned above, the first pixel position used to characterize the location of the target object in the image can include one or more pixel positions. For example, in... Figure 2A In this process, the set of pixels corresponding to the target object in the image can be determined based on the fixed shape and size of the target object and the position of the first pixel. For example, in... Figure 2B In this context, based on the pixel positions of the four vertices included in the first pixel position, the set of pixels enclosed by the rectangle formed by these four vertices can be determined as the pixel set corresponding to the target object in the image. For example, in... Figure 2C In this process, the pixel set corresponding to the target object in the image can be determined by the pixel positions of the contours included by the first pixel position, encompassing the region enclosed by the contours. Alternatively, the pixel set corresponding to the bounding boxes included by the first pixel position can be determined as the pixel set corresponding to the target object in the image. Then, the point cloud corresponding to the pixel set is determined from the point cloud as the target point cloud.

[0078] In other embodiments, when determining the target point cloud that matches the first pixel position, the range corresponding to the first pixel position can be determined from the point cloud, and the point cloud included in the range can be determined as the target point cloud, that is, the point cloud of the target object.

[0079] As in Figure 2A In this process, based on the fixed shape and size of the target object and the position of the first pixel, a corresponding range can be determined from the point cloud, and the point cloud encompassed by this range can be defined as the target point cloud. For example, in... Figure 2B In this process, a rectangular area corresponding to four vertices can be determined from the point cloud, and the point cloud within this rectangular area can be identified as the target point cloud. For example, in... Figure 2C In this process, the range corresponding to the pixel position of the contour can be determined from the point cloud, and the point cloud included in this range can be identified as the target point cloud.

[0080] The point cloud processing method provided in this specification can be applied to any scenario requiring point cloud annotation, including but not limited to high-precision map construction. Through the above steps, the process of annotating the target object in the point cloud is completed. Since images acquired by image acquisition devices are more vivid and have higher recognizability than point cloud data, using images with annotated target objects to assist in point cloud annotation can effectively reduce the difficulty of direct annotation in the point cloud, improve the accuracy of annotation, and enhance the accuracy of the constructed high-precision map.

[0081] In some embodiments, the field of view of the point cloud acquisition device is larger than and includes the field of view of the image acquisition device. That is, the acquisition range of the point cloud is larger than and includes the acquisition range of the image. Thus, a third point cloud can be segmented from a frame of point cloud acquired by the point cloud acquisition device based on the relative positional relationship between the point cloud acquisition device and the image acquisition device, as well as the size of the image. Then, step 130 can be used to determine the target point cloud that matches the position of the first pixel from the segmented third point cloud. Determining the third point cloud corresponding to the image from the point cloud with a larger acquisition range first can reduce the computational complexity when determining the target point cloud and help speed up the matching speed of the target point cloud.

[0082] Regarding the annotation process of target objects in images, in some embodiments, semantic segmentation processing can be performed on the image to automatically identify the target object from the image, thereby obtaining the first pixel position of the target object in the image. Semantic segmentation of images can be found in related technical descriptions, and will not be elaborated upon here. The annotation of the target object and the point cloud processing method provided in the embodiments of this specification can be executed by the same computing device or by different computing devices. For example, after completing the annotation of the target object, one computing device can send the obtained first pixel position to another computing device, so that the other computing device can execute the aforementioned point cloud processing method.

[0083] In other embodiments, a labeling tool can be used to manually label the target object in the image. After manually labeling the target object, the obtained first pixel position can be stored in a storage device. When performing step 120, the pre-stored first pixel position can be retrieved from the storage device, that is, the pre-stored first pixel position after manually labeling the target object in the image can be retrieved.

[0084] Of course, you can also perform semantic segmentation on the image first to automatically identify the target object in the image, and then correct the identification results and adjust the position of the first pixel through manual annotation to improve the annotation accuracy of the target object.

[0085] Furthermore, in addition to marking the first pixel position of the target object in the image through the aforementioned semantic segmentation and / or manual annotation, the type of the target object can also be labeled. There are various ways to classify target object types; for example, target object types can include ground feature types and non-ground feature types. For instance, ground feature types can be further divided into lane line types, arrow types, stop line types, etc., while non-ground feature types can be further divided into building types, sign types, pole types, etc. Moreover, after determining the target point cloud from the point cloud, a type label can be added to the target point cloud based on the type of the target object for subsequent processing.

[0086] In related technologies, annotation is often done manually or by combining semantic segmentation with manual annotation directly on point clouds. During manual annotation, because point clouds are three-dimensional data with low resolution, technicians often need to repeatedly compare the point cloud to be annotated with its corresponding two-dimensional image, constantly adjusting the annotation range to obtain a relatively accurate point cloud annotation. This undoubtedly brings a significant workload to technicians and affects the efficiency of point cloud annotation. However, in the method provided in the embodiments of this specification, technicians can directly annotate the target object in an image with high resolution, obtaining the first pixel position of the target object in the image. Then, based on the relative positional relationship between the image acquisition device and the point cloud acquisition device, the target point cloud of the target object matching the first pixel position is determined from the point cloud. On the one hand, since technicians can directly annotate in an image with high resolution, there is no need to repeatedly compare the point cloud with the image, reducing the workload of technicians and thus greatly improving the efficiency of point cloud annotation; on the other hand, compared with direct manual annotation on point clouds, the point cloud annotated by the method provided in the embodiments of this specification has higher accuracy.

[0087] Images acquired by the image acquisition device and point clouds acquired by the point cloud acquisition device are both raw data, often stored in a database and retrieved from the database when needed. Generally, when performing computational tasks using raw data, the raw data is not modified to ensure its integrity. Thus, in some embodiments, the process of determining the target point cloud in step 130 above may include, for example... Figure 3 The steps shown are as follows:

[0088] Step 310: Based on the first pixel position, generate a mask image corresponding to the image;

[0089] The mask image is marked with the position of the first pixel and is used to remove other image content except for the target object;

[0090] Step 320: Based on the relative positional relationship between the point cloud acquisition device and the image acquisition device, project the point cloud onto the mask image;

[0091] Step 330: Determine the target point cloud of the target object that matches the position of the first pixel from the point cloud.

[0092] After obtaining the first pixel position, a mask image can be generated for the image object. The first pixel position can be marked on the mask image, retaining the image content corresponding to the target object while removing other image content. Thus, only the pixel region corresponding to the target object participates in subsequent calculations.

[0093] Subsequently, based on the relative positions of the point cloud acquisition device and the image acquisition device, the point cloud can be projected onto a mask image, and the target point cloud matching the first pixel position in the mask image is identified as the point cloud of the target object. By generating a mask image and projecting the point cloud onto it to determine the target point cloud, on the one hand, since the mask image only retains the image content corresponding to the target object, the amount of data processing in subsequent calculations is reduced. On the other hand, using the mask image in the calculation avoids modification of the original data (image) during the calculation process, maintaining the originality of the image.

[0094] The image acquisition device and the point cloud acquisition device acquire data according to a preset acquisition cycle. During the data acquisition process, data on the same target object may be acquired in several acquisition cycles. That is, several frames of data acquired within a certain period of time may all include data on the same target object. However, due to fluctuations in the acquisition device or occlusion of the target object, data from some acquisition cycles may be missing. Therefore, in order to improve the accuracy of target object annotation, in some embodiments, the target point cloud determination process in step 130 above may include, for example... Figure 4 The steps shown are as follows:

[0095] Step 410: Acquire multiple frames of point clouds collected within a preset time period, and determine the point cloud that matches the position of the first pixel from each frame of the point cloud;

[0096] Step 420: Perform fusion processing on the point clouds matched in each frame to obtain the target point cloud of the target object.

[0097] The fusion processing of the point clouds matched in each frame may include determining the union of the point clouds matched in multiple frames as the point cloud of the target object. In this way, by fusing the point clouds matched in multiple frames, point cloud data missing in certain acquisition cycles can be supplemented using point cloud data from other acquisition cycles. This improves the completeness of the point cloud of the target object and the accuracy of point cloud annotation.

[0098] The point cloud processing method provided in any of the above embodiments can be applied in the construction of high-precision maps. Thus, the target object can be any type of road feature. As mentioned above, road features include ground features on the ground and non-ground features at a certain distance from the ground. In some embodiments, if the target object is a ground feature, such as lane lines, stop lines, yield lines, ground arrows, text, guide lines, etc., the process of determining the target point cloud in step 130 above may include, for example... Figure 5 The steps shown are as follows:

[0099] Step 510: Obtain the second pixel position of the target object in the projected image;

[0100] The projected image is an image obtained by projecting the two-dimensional point cloud onto the ground after identifying the two-dimensional point cloud from the point cloud.

[0101] Step 520: Based on the relative positional relationship, determine a first point cloud from the two-dimensional point cloud that matches the position of the first pixel;

[0102] Step 530: Based on the coordinate transformation relationship between the two-dimensional point cloud and the projected image, determine the second point cloud that matches the position of the second pixel from the two-dimensional point cloud;

[0103] Step 540: Determine the target point cloud based on the first point cloud and the second point cloud.

[0104] In this point cloud, each point carries its three-dimensional position information. This three-dimensional position information includes the point's elevation information, that is, its height relative to sea level. Thus, the generation of the projected image involves... Figure 6 The steps shown are as follows:

[0105] Step 610: Determine the point clouds whose elevation information is less than a preset threshold from the point clouds as the two-dimensional point clouds;

[0106] Step 620: Generate the projected image based on the two-dimensional position information of the two-dimensional point cloud;

[0107] The two-dimensional location information is obtained based on the three-dimensional location information of the two-dimensional point cloud.

[0108] Each point in a point cloud typically carries its own three-dimensional position information. This three-dimensional position information can be represented by three-dimensional coordinates (x, y, z). In the raw data collected by the point cloud acquisition device, the three-dimensional position information carried by the point cloud is the relative three-dimensional position information of the points with respect to the acquisition device. During point cloud acquisition, the geographic location information of the map acquisition vehicle can be obtained through a positioning device (such as GNSS or IMU) mounted on the vehicle, and the geographic location information of the point cloud acquisition device can be determined based on its position on the vehicle. Thus, the geographic location information of each point can be determined based on the geographic location information of the acquisition device and the relative three-dimensional position information carried by each point. After the above preprocessing of the point cloud data, the three-dimensional position information carried by the points in the point cloud can be considered geographic location information. Simultaneously, based on the calibration height of the point cloud acquisition device relative to the ground, the elevation information of each point can be determined from its three-dimensional position information.

[0109] Thus, by comparing the elevation information of each point with a preset threshold, it can be determined that point clouds with elevation information less than the preset threshold are ground point clouds, i.e., two-dimensional point clouds, including point clouds of ground features. Point clouds with elevation information greater than the preset threshold are point clouds of three-dimensional objects above the ground, i.e., three-dimensional point clouds, including point clouds of non-ground features.

[0110] After determining the 2D point cloud, 2D position information can be obtained based on its 3D position information. For example, position information other than elevation information in the 3D position information can be considered 2D position information. For instance, 2D coordinates (x, y) can be determined based on 3D coordinates (x, y, z). Obtaining 2D position information from 3D position information is equivalent to projecting the 2D point cloud onto the ground to obtain a projected image. The 2D position information can then be used to determine the position of each point in the 2D point cloud within the projected image.

[0111] As an example, the two-dimensional coordinates (x, y) can be determined as the pixel coordinates (u, v) of the point in the projected image. As another example, the two-dimensional coordinates (x, y) can be mapped to the pixel coordinates (u, v) through an affine transformation. Affine transformations can be found in related art descriptions, and will not be elaborated upon here.

[0112] In some embodiments, points in the point cloud can also carry visual information, including color information and / or reflectance intensity information. Differences in the materials of different objects result in varying reflectance intensities in the point cloud; therefore, points reflecting from different objects carry different reflectance intensity information. Furthermore, since the image carries the true color information of the object, a color image carrying three-dimensional position information can be used to colorize the point cloud, ensuring that each point in the point cloud carries true color information. Thus, the projected image can also be generated based on the two-dimensional position information and visual information of the two-dimensional point cloud. In some examples, the projected image can be a one-channel grayscale image or a three-channel color image. In some examples, the alpha channel of each pixel in the projected image can also carry the elevation information of the corresponding point in the two-dimensional point cloud for subsequent processing.

[0113] After obtaining the projected image, the target object can be marked from the projected image, and the second pixel position of the target object in the projected image can be obtained. The marked target object includes ground features. The process of marking the target object in the projected image can refer to the process of marking the target object in the image described above, and the process of obtaining the second pixel position can refer to the process of obtaining the first pixel position described above; these will not be repeated here. The marking of the target object in the projected image and the point cloud processing method provided in the embodiments of this specification can be performed by the same computing device or by different computing devices.

[0114] Thus, the same target object is labeled in both the image acquired by the image acquisition device and the projected image of the 2D point cloud. Therefore, point cloud labeling can be performed simultaneously using both the image and the projected image. Specifically, this can include, for example... Figure 7 The steps shown are as follows:

[0115] Step 710: Obtain the second pixel position of the target object in the projected image;

[0116] Step 720: Based on the relative positional relationship, determine a first point cloud from the two-dimensional point cloud that matches the position of the first pixel;

[0117] Step 730: Based on the coordinate transformation relationship between the two-dimensional point cloud and the projected image, determine the second point cloud that matches the position of the second pixel from the two-dimensional point cloud;

[0118] Step 740: Perform a fusion process on the first point cloud and the second point cloud to obtain the target point cloud.

[0119] In determining the second point cloud from the 2D point cloud that matches the second pixel position, the corresponding pixel set of the target object in the projected image can be determined at least based on the second pixel position. Then, the point cloud corresponding to the pixel set is determined as the target point cloud. The process of determining the pixel set of the target object in the projected image using the second pixel position is similar to the process described above of determining the pixel set of the target object in the image using the first pixel position. Furthermore, when determining the second point cloud, a mask image corresponding to the projected image can be generated based on the second pixel position, and the point cloud can be projected onto the mask image to determine the second point cloud. See the above for details. Figure 3 The corresponding embodiments are not described in detail here.

[0120] The fusion process of the first point cloud and the second point cloud may include: determining the union or intersection of the first point cloud and the second point cloud as the target point cloud.

[0121] In this embodiment, a two-dimensional point cloud is extracted from the point cloud, and then the two-dimensional point cloud is projected onto the ground to obtain a projected image. Both the projected image and the image acquired by the image acquisition device can be labeled with the same ground feature as the target object. Combining the labeling of the same target object from both images assists in labeling the target object in the point cloud, further reducing the difficulty of direct labeling in the point cloud and improving the accuracy of the labeling and the accuracy of the constructed high-precision map.

[0122] As described above, in the raw data collected by the point cloud acquisition device, each point in the point cloud carries its relative position information with respect to the acquisition device. Therefore, based on any of the above embodiments, a point cloud processing method can be used to construct or correct a map using the determined target point cloud. Map correction refers to revising an already constructed map. In an already constructed map, errors in image semantic segmentation may lead to incorrect or missing labeling of target objects. In this case, target objects can be manually labeled on the image, thereby re-determining the point cloud of the target objects and correcting the map. Taking map construction using the target point cloud as an example, the map construction process may include, for example... Figure 8 The steps shown are as follows:

[0123] Step 810: Based on the geographic location information recorded by the point cloud acquisition device when acquiring the point cloud, and the relative position information of the target point cloud, obtain the geographic location information of the target object;

[0124] Step 820: Construct a map based on the geographic location information of the target object.

[0125] As mentioned above, since the map acquisition vehicle is equipped with positioning devices such as GNSS and IMU, during the point cloud acquisition process, the geographical location information of the map acquisition vehicle obtained by the positioning device, as well as the setting position of the point cloud acquisition device on the map acquisition vehicle, can be used to determine the geographical location information of the point cloud acquisition device when acquiring each frame of point cloud. Thus, based on the geographical location information of the point cloud acquisition device when acquiring the target point cloud and the relative position information of the target point cloud, the geographical location information of the target object can be determined. Then, based on the geographical location information of the target object, a map can be constructed. For example, this map can be a high-precision map. As an example, the marked target objects can be vectorized to obtain corresponding vector data. The vectorization process refers to expressing the spatial characteristics of ground features using points, lines, surfaces, or three-dimensional geometric figures. For example, if the shape of the target object is a straight line, such as a lane line, then vector data (point 1, point 2) can be used to represent the target object. Point 1 carries three-dimensional spatial information, such as three-dimensional coordinates (X, Y, Z). Subsequently, a map can be constructed using the vector data of the target object. Specific vectorization and map construction processes can be found in relevant technologies, and will not be discussed in detail here.

[0126] As mentioned above, when labeling target objects in an image, the type of the target object can also be labeled. Thus, in some embodiments, during map construction, the visualization model corresponding to the target object can be determined from a visualization model library based on the target object's type. Then, a map is constructed based on the target object's geographic location information and the visualization model. The visualization model library stores visualization models corresponding to different types of objects, thereby providing a visual representation of different types of objects displayed on the map. The corresponding visualization model can be determined from the visualization model library based on the target object's type.

[0127] In other embodiments, the shape information of the target object can be obtained based on the target point cloud. Then, a map is constructed based on the geographical location information and shape information of the target object. Shape information may include contour information and size information, etc.

[0128] Of course, different target objects of the same type may differ in size; for example, two rectangular traffic signs may differ in length and width. Therefore, we can first obtain the corresponding visualization model based on the type of the target object, and then adjust the parameters used to control the size in the visualization model based on the shape information determined by the target point cloud, so that the adjusted visualization model matches the size of the target object.

[0129] This specification provides a point cloud processing method that acquires a point cloud collected by a point cloud acquisition device and an image acquired by an image acquisition device, wherein the acquisition range of the image matches the acquisition range of the point cloud. The method obtains the first pixel position of a target object in the image, and based on the relative positional relationship between the two acquisition devices, determines the target point cloud that matches the first pixel position as the point cloud of the target object, thereby completing the process of marking the target object in the point cloud. Since the image acquired by the image acquisition device is more vivid and has higher recognizability than point cloud data, using the image with the marked target object to assist the point cloud in marking the target object can effectively reduce the difficulty of direct marking in the point cloud, improve the accuracy of marking, and enhance the accuracy of the constructed high-precision map.

[0130] In addition, this application also provides a point cloud processing method, including as follows: Figure 9 The steps shown are as follows:

[0131] Step 910: Acquire the point cloud acquired by the point cloud acquisition device and the image acquired by the image acquisition device at the same time;

[0132] The acquisition range of the image matches the acquisition range of the point cloud. Each point in the point cloud carries its three-dimensional position information, including its elevation information.

[0133] Step 920: Based on the relative positional relationship between the point cloud acquisition device and the image acquisition device, and the size of the image, determine the point cloud corresponding to the image from the point cloud.

[0134] Step 931: Obtain the first pixel position of the target object in the image after semantic segmentation or manual annotation, as well as the type of the target object;

[0135] The target object types include ground feature types and non-ground feature types.

[0136] Step 932: Generate a mask image corresponding to the image based on the first pixel position, project the point cloud onto the mask image to obtain the first point cloud matching the first pixel position, and label the first point cloud with a type according to the type of the target object;

[0137] The mask image is marked with the position of the first pixel and is used to remove image content other than the target object. If the type label of the first point cloud is a non-ground feature type label, then step 951 is executed; if the type label of the first point cloud is a ground feature type label, then step 952 is executed.

[0138] Step 940: Identify point clouds with elevation information less than a preset threshold as two-dimensional point clouds, and generate a projection image based on the two-dimensional position information of the two-dimensional point clouds;

[0139] Step 941: Obtain the second pixel position of the target object in the projected image after semantic segmentation or manual annotation, as well as the type of the target object;

[0140] The types of target objects include ground feature types.

[0141] Step 942: Generate a mask image corresponding to the projected image based on the second pixel position, project the point cloud onto the mask image to obtain a second point cloud that matches the second pixel position; and label the second point cloud with a type according to the type of the target object.

[0142] The mask image is marked with the second pixel position, which is used to remove image content other than the target object. Then, step 952 is executed.

[0143] Step 951: If the type label of the first point cloud is a non-ground feature type label, the first point cloud determined by each frame of point cloud collected by the point cloud acquisition device within the preset time period is fused to obtain the target point cloud of the target object.

[0144] Step 952: If the type label of the first point cloud is the label of the ground feature type, the first point cloud and the second point cloud determined by each frame of point cloud collected by the point cloud acquisition device within the preset time period are fused to obtain the target point cloud of the target object.

[0145] In other words, for a target object of the ground feature type, a first point cloud and a second point cloud can be determined from a single frame of point cloud acquired by the point cloud acquisition device. Fusing the first and second point clouds yields a single frame of fused point cloud. By fusing the first and second point clouds in each frame of point cloud acquired by the point cloud acquisition device within a preset time period, multiple frames of fused point cloud can be obtained. Further fusing these multiple frames of fused point cloud yields the target point cloud of the target object.

[0146] Step 960: Based on the geographical location information recorded when the point cloud is collected by the point cloud acquisition device, and the relative position information of the target point cloud, obtain the geographical location information of the target object; construct a map based on the geographical location information of the target object.

[0147] The specific implementation methods of the above steps are described in the embodiments above, and will not be repeated here. Furthermore, the execution order of steps 931-932 and 940-942 is not sequential and they can be executed simultaneously. The semantic segmentation processing in steps 931 and 941, and the point cloud processing method of this embodiment, can be executed by the same computing device or by different computing devices.

[0148] Based on the point cloud processing method described in any of the above embodiments, this specification also provides, as follows: Figure 10 The diagram shows the structure of an electronic device. Figure 10 At the hardware level, the electronic device includes a processor, an internal bus, a network interface, memory, and non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to implement a point cloud processing method as described in any of the above embodiments.

[0149] Based on the point cloud processing method described in any of the above embodiments, this specification also provides a point cloud processing apparatus, including:

[0150] The first acquisition module is used to acquire point clouds acquired by the point cloud acquisition device and images acquired by the image acquisition device; wherein the acquisition range of the image matches the acquisition range of the point cloud.

[0151] The second acquisition module is used to acquire the first pixel position of the target object in the image;

[0152] The determining module is used to determine, at least based on the relative positional relationship between the point cloud acquisition device and the image acquisition device, the target point cloud of the target object that matches the position of the first pixel from the point cloud.

[0153] The specific implementation methods of each of the above functional modules are described in the above embodiments, and will not be repeated here.

[0154] Based on the point cloud processing method described in any of the above embodiments, this specification also provides a computer program product, including a computer program, which, when executed by a processor, can be used to execute the point cloud processing method described in any of the above embodiments.

[0155] Based on the point cloud processing method described in any of the above embodiments, this specification also provides a computer storage medium storing a computer program, which, when executed by a processor, can be used to execute the point cloud processing method described in any of the above embodiments.

[0156] The foregoing has described specific embodiments of the embodiments described in this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0157] Other embodiments of the embodiments of this specification will readily occur to those skilled in the art upon consideration of the specification and practice of the invention claimed herein. The embodiments of this specification are intended to cover any variations, uses, or adaptations of the embodiments of this specification that follow the general principles of the embodiments of this specification and include common knowledge or customary techniques in the art not claimed in the embodiments of this specification. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of the embodiments of this specification are indicated by the following claims.

Claims

1. A point cloud processing method, characterized in that, The method includes: The method acquires point clouds collected by a point cloud acquisition device and images collected by an image acquisition device; wherein the acquisition range of the image matches the acquisition range of the point cloud. Obtain the first pixel position of the marked target object in the image; Based at least on the pre-defined relative positional relationship between the point cloud acquisition device and the image acquisition device, a target point cloud of the target object matching the first pixel position is determined from the point cloud, in order to mark the target object in the point cloud, including: Obtain the second pixel position of the target object in the projected image; wherein, the projected image is the image after projecting the two-dimensional point cloud onto the ground after identifying points in the point cloud whose elevation information is less than a preset threshold. Based on the relative positional relationship, a first point cloud that matches the position of the first pixel is determined from the two-dimensional point cloud; Based on the coordinate transformation relationship between the two-dimensional point cloud and the projected image, a second point cloud matching the position of the second pixel is determined from the two-dimensional point cloud; The target point cloud is determined based on the first point cloud and the second point cloud.

2. The method according to claim 1, characterized in that, The points in the point cloud carry the three-dimensional position information of the points; the three-dimensional position information includes the elevation information of the points. After identifying the two-dimensional point cloud from the point cloud, projecting the two-dimensional point cloud onto the ground includes: The point cloud whose elevation information is less than a preset threshold is identified as the two-dimensional point cloud. The projected image is generated based on the two-dimensional position information of the two-dimensional point cloud; wherein the two-dimensional position information is obtained based on the three-dimensional position information of the two-dimensional point cloud.

3. The method according to claim 1, characterized in that, Determining the target point cloud based on the first point cloud and the second point cloud includes: The first point cloud and the second point cloud are fused to obtain the target point cloud.

4. The method according to claim 1, characterized in that, The step of determining the target point cloud of the target object matching the first pixel position from the point cloud, at least based on the pre-calibrated relative positional relationship between the point cloud acquisition device and the image acquisition device, includes: Based on the relative positional relationship and the size of the image, a third point cloud is segmented from a frame of point cloud acquired by the point cloud acquisition device; Based on the relative positional relationship, the target point cloud of the target object that matches the position of the first pixel is determined from the third point cloud.

5. The method according to claim 1, characterized in that, Obtaining the first pixel position of the labeled target object in the image includes: Perform semantic segmentation on the image to obtain the first pixel position of the target object in the image; and / or Obtain the first pixel position stored in the image after the target object is manually marked.

6. The method according to claim 1, characterized in that, The step of determining the target point cloud of the target object matching the first pixel position from the point cloud, at least based on the pre-calibrated relative positional relationship between the point cloud acquisition device and the image acquisition device, includes: Based on the first pixel position, a mask image corresponding to the image is generated; the mask image is marked with the first pixel position and is used to remove other image content except for the target object; Based on the relative positional relationship between the point cloud acquisition device and the image acquisition device, the point cloud is projected onto the mask image; The target point cloud of the target object that matches the position of the first pixel is determined from the point cloud.

7. The method according to claim 1, characterized in that, Determining the target point cloud of the target object that matches the first pixel position from the point cloud includes: Acquire multiple frames of point clouds collected within a preset time period, and determine the point cloud that matches the position of the first pixel from each frame of the point cloud; The point clouds matched in each frame are fused together to obtain the target point cloud of the target object.

8. The method according to claim 1, characterized in that, The points in the point cloud carry relative position information of the points relative to the point cloud acquisition device; the method further includes: Based on the geographic location information recorded by the point cloud acquisition device when acquiring the point cloud, and the relative position information of the target point cloud, the geographic location information of the target object is obtained; A map is constructed based on the geographic location information of the target object.

9. The method according to claim 8, characterized in that, The process of constructing a map based on the geographic location information of the target object includes: Based on the type of the target object, a visualization model corresponding to the target object is determined from a visualization model library; wherein, the visualization model library stores visualization models corresponding to different types of objects; a map is constructed based on the geographic location information of the target object and the visualization model; and / or Based on the target point cloud, obtain the shape information of the target object; construct a map based on the geographical location information of the target object and the shape information.

10. An electronic device, characterized in that, The electronic device includes: processor; Memory used to store processor-executable instructions; Wherein, when the processor invokes the executable instructions, it implements the operation of any one of the methods described in claims 1-9.

11. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the method as claimed in any one of claims 1-9.

Images