Element extraction and map generation method and device based on zero sample deep learning

Abstract

The invention provides an element extraction and map generation method and device based on zero sample deep learning, and the method comprises the steps: carrying out the intensity projection of a three-dimensional point cloud image which needs to be subjected to road element extraction, and obtaining a two-dimensional intensity projection image; inputting the two-dimensional intensity projection image into a road element deep learning model to obtain key point two-dimensional coordinate information of the road elements output by the road element deep learning model; the road element deep learning model sequentially comprises a YOLOv8-OBB rotating frame model, a calculation module comprising a PCA algorithm, and a multi-head internet key point detection model; according to the reverse logic of the processing logic of the road element deep learning model, determining the two-dimensional coordinate information of the two-dimensional coordinate information of the key point on the two-dimensional intensity projection image; and finally, determining the three-dimensional coordinates of the road elements on the three-dimensional point cloud image according to the reverse logic of the processing logic of the intensity projection and the two-dimensional coordinate information of the two-dimensional coordinate information of the key points on the two-dimensional intensity projection image.

Description

Technical Field

[0001] The present disclosure relates to the field of autonomous driving technology, and in particular to the field of map construction technology, and discloses a method and device for element extraction and map generation based on zero-sample deep learning. Background Art

[0002] With the development of autonomous driving technology, road elements are not only signs to ensure the safety of pedestrians and vehicles, but also an indispensable basic element in high-precision maps that meet the needs of autonomous driving. Existing road element extraction technologies are mainly divided into image-based detection methods and point cloud-based detection methods.

[0003] The main advantage of image-based detection methods is that images have high resolution and rich texture information, and the detection effect is relatively ideal. However, image-based detection methods have the following shortcomings:

[0004] Strong environmental dependence: Image detection is greatly affected by external conditions such as lighting and weather, resulting in poor robustness; Lack of spatial information: Two-dimensional image detection methods cannot directly obtain three-dimensional geometric information, resulting in the inability to provide accurate three-dimensional positioning when building high-precision maps; High labor costs: There are certain differences in road elements in various countries, and the labor cost of producing different data sets is high.

[0005] In the point cloud-based detection method, the three-dimensional point cloud data generated by the LiDAR contains rich spatial information and can more accurately represent the physical location of road features. However, point cloud detection also has shortcomings:

[0006] High computing cost: Processing large amounts of point cloud data requires high computing resources, especially in real-time applications, and it is difficult to complete the processing efficiently; Data sparsity: Due to the working method of lidar, especially at long distances or in complex terrain, point cloud data is often sparse and incomplete, affecting the accuracy of feature extraction. Summary of the invention

[0007] The present disclosure at least provides a method and device for feature extraction and map generation based on zero-sample deep learning, which solves at least one of the problems in actual production, such as low accuracy of road feature extraction, poor robustness, difficulty in projecting into 3D point cloud data, high labor cost for data set production, and poor real-time performance.

[0008] According to one aspect of the present disclosure, a method for extracting features based on zero-sample deep learning is provided, comprising:

[0009] Obtain a pre-trained road feature deep learning model and a 3D point cloud image that requires road feature extraction;

[0010] Performing intensity projection on the three-dimensional point cloud image to obtain a two-dimensional intensity projection image;

[0011] Inputting the two-dimensional intensity projection image into the road element deep learning model to obtain the two-dimensional coordinate information of the key points of the road elements output by the road element deep learning model; the road element deep learning model includes a YOLOv8-OBB rotating box model, a computing module including a PCA algorithm, and a multi-head centernet key point detection model in sequence;

[0012] Determine the two-dimensional coordinate information of the key point two-dimensional coordinate information on the two-dimensional intensity projection image according to the reverse logic of the processing logic of the road element deep learning model;

[0013] Determine the three-dimensional coordinates of the road element on the three-dimensional point cloud image according to the reverse logic of the processing logic of the intensity projection and the two-dimensional coordinate information of the key point on the two-dimensional intensity projection image;

[0014] The data set used to train the multi-head centernet key point detection model is obtained according to the following steps:

[0015] Acquire multiple three-dimensional point cloud sample images, and perform intensity projection on each of the three-dimensional point cloud sample images to obtain multiple two-dimensional intensity sample projection images;

[0016] Determine a plurality of road element identification templates and key points corresponding to each road element identification template by using each two-dimensional intensity sample projection image;

[0017] Fill each road element identification template with the first target scene color, and rotate them at least once according to the first preset angle range to obtain multiple first road element template images, and use the data set formed by the first road element template images as the data set for training the multi-head centernet key point detection model;

[0018] The data set used to train the YOLOv8-OBB rotating box model is obtained according to the following steps:

[0019] Filling each road element identification template with a second target scene color, and rotating each of the road element identification templates at least once according to a second preset angle range, to obtain a plurality of second road element template images; pasting the obtained second road element template images onto at least one road background image according to at least one randomly determined position, to obtain a plurality of training sample images;

[0020] Using the data set formed by the training sample images as a data set for training the YOLOv8-OBB rotating box model;

[0021] Among them, the processing logic of the road element deep learning model is as follows:

[0022] The YOLOv8-OBB rotating frame model is used to extract the rotating frame position information of the road element from the two-dimensional intensity projection image; the rotating frame position information is two-dimensional coordinate information;

[0023] Use the PCA algorithm to perform principal component analysis on the rotation frame position information of road elements to obtain the main direction of the rotation frame;

[0024] According to the angle between the main direction of the rotation frame and the vertical direction, the pixel points corresponding to the road elements on the two-dimensional intensity projection image are rotated;

[0025] The multi-head CenterNet key point detection model is used to extract the key points of the rotated road elements to obtain the two-dimensional coordinate information of the key points.

[0026] In a possible implementation, performing intensity projection on the three-dimensional point cloud image to obtain a two-dimensional intensity projection image includes:

[0027] Determine the width and height of the point cloud according to the position information of each point in the three-dimensional point cloud image;

[0028] Mapping each point in the three-dimensional point cloud image to an image grid according to the width and height of the point cloud to obtain a position index of each point in the two-dimensional intensity projection image;

[0029] For an image grid with only one point, the intensity of the point in the image grid is filled into the corresponding image network; for an image grid with multiple point clouds, the maximum intensity is filled into the corresponding image network.

[0030] In a possible implementation, determining the width and height of the point cloud according to the position information of each point in the three-dimensional point cloud image includes:

[0031] Determine the maximum and minimum values ​​of the point cloud in the x direction according to the position information of each point in the three-dimensional point cloud image;

[0032] Determine the width of the point cloud according to the maximum value of the point cloud in the x direction, the minimum value of the point cloud in the x direction, and the preset resolution;

[0033] Determine the maximum and minimum values ​​of the point cloud in the y direction according to the position information of each point in the three-dimensional point cloud image;

[0034] Determine the height of the point cloud according to the maximum value of the point cloud in the y direction, the minimum value of the point cloud in the y direction, and the preset resolution;

[0035] The x direction and the y direction are perpendicular to each other.

[0036] In a possible implementation, mapping each point in the three-dimensional point cloud image to an image grid according to the width and height of the point cloud to obtain a position index of each point in the two-dimensional intensity projection image includes:

[0037] For each point, the first index of the point in the y direction is determined according to the coordinate of the point in the y direction, the maximum value of the point cloud in the y direction and the preset resolution; the second index of the point in the x direction is determined according to the coordinate of the point in the x direction, the maximum value of the point cloud in the x direction and the preset resolution; the first index and the second index are used as the position index of the point in the two-dimensional intensity projection image.

[0038] In a possible implementation, the using of the PCA algorithm to perform principal component analysis on the rotation frame position information of the road element to obtain the main direction of the rotation frame includes:

[0039] Determine the two-dimensional coordinates of the centroid of the rotating frame according to the rotating frame position information of the road element;

[0040] Decentralizing each boundary point of the rotation frame using the two-dimensional coordinates of the centroid to obtain a decentralized point corresponding to each boundary point;

[0041] Using the coordinates of each decentralized point, determine the covariance matrix;

[0042] Perform eigenvalue decomposition on the covariance matrix, and determine the main direction of the rotating frame according to the result of the eigenvalue decomposition.

[0043] In a possible implementation, the decentralized processing of each boundary point of the rotation frame using the two-dimensional coordinates of the centroid to obtain decentralized points corresponding to each boundary point includes:

[0044] For each boundary point, the decentralization process is performed using the following formula:

[0045] P′ i =(x i -C x ,y i -C y )

[0046] Among them, (C x , C y ) represents the two-dimensional coordinates of the center of mass, (x i ,y i ) represents the coordinates of the boundary point i, P′ i Represents the decentralized point obtained by decentralized processing.

[0047] In a possible implementation manner, the first target scene color and / or the second target scene color is white.

[0048] According to another aspect of the present disclosure, a device for extracting elements based on zero-sample deep learning is provided, comprising:

[0049] A data acquisition module is used to obtain a pre-trained road element deep learning model and a three-dimensional point cloud image that needs to be extracted from the road element;

[0050] An intensity projection module, used for performing intensity projection on the three-dimensional point cloud image to obtain a two-dimensional intensity projection image;

[0051] A key point extraction module, used for inputting the two-dimensional intensity projection image into the road element deep learning model to obtain the two-dimensional coordinate information of the key points of the road elements output by the road element deep learning model; the road element deep learning model includes a YOLOv8-OBB rotating box model, a calculation module including a PCA algorithm, and a multi-head centernet key point detection model in sequence;

[0052] A first reverse logic processing module, used to determine the two-dimensional coordinate information of the key point two-dimensional coordinate information on the two-dimensional intensity projection image according to the reverse logic of the processing logic of the road element deep learning model;

[0053] A second reverse logic processing module, used for determining the three-dimensional coordinates of the road element on the three-dimensional point cloud image according to the reverse logic of the processing logic of the intensity projection and the two-dimensional coordinate information of the key point on the two-dimensional intensity projection image;

[0054] Among them, the key point extraction module is also used to determine the data set used to train the multi-head centernet key point detection model:

[0055] Acquire multiple three-dimensional point cloud sample images, and perform intensity projection on each of the three-dimensional point cloud sample images to obtain multiple two-dimensional intensity sample projection images;

[0056] Determine a plurality of road element identification templates and key points corresponding to each road element identification template by using each two-dimensional intensity sample projection image;

[0057] Fill each road element identification template with the first target scene color, and rotate them at least once according to the first preset angle range to obtain multiple first road element template images, and use the data set formed by the first road element template images as the data set for training the multi-head centernet key point detection model;

[0058] Among them, the key point extraction module is also used to determine the data set for training the YOLOv8-OBB rotating box model:

[0059] Filling each road element identification template with a second target scene color, and rotating each of the road element identification templates at least once according to a second preset angle range, to obtain a plurality of second road element template images; pasting the obtained second road element template images onto at least one road background image according to at least one randomly determined position, to obtain a plurality of training sample images;

[0060] Using the data set formed by the training sample images as a data set for training the YOLOv8-OBB rotating box model;

[0061] Among them, the processing logic of the road element deep learning model is as follows:

[0062] The YOLOv8-OBB rotating frame model is used to extract the rotating frame position information of the road element from the two-dimensional intensity projection image; the rotating frame position information is two-dimensional coordinate information;

[0063] Use the PCA algorithm to perform principal component analysis on the rotation frame position information of road elements to obtain the main direction of the rotation frame;

[0064] According to the angle between the main direction of the rotation frame and the vertical direction, the pixel points corresponding to the road elements on the two-dimensional intensity projection image are rotated;

[0065] The multi-head CenterNet key point detection model is used to extract the key points of the rotated road elements to obtain the two-dimensional coordinate information of the key points.

[0066] According to another aspect of the present disclosure, a map generation method is provided, comprising:

[0067] Determine the three-dimensional coordinates of each road element in the three-dimensional point cloud image on the three-dimensional point cloud image using any of the above-mentioned zero-sample deep learning-based feature extraction methods;

[0068] A map including road elements is generated based on the determined three-dimensional coordinates.

[0069] According to another aspect of the present disclosure, there is provided a map generating device, comprising:

[0070] A three-dimensional coordinate determination module, used to determine the three-dimensional coordinates of each road element in the three-dimensional point cloud image on the three-dimensional point cloud image using any of the above-mentioned feature extraction methods based on zero-sample deep learning;

[0071] The map generation module is used to generate a map including road elements according to the determined three-dimensional coordinates.

[0072] The disclosed method and device for extracting features and generating maps based on zero-sample deep learning adopts a zero-sample customized deep learning method, and automatically generates a data set used for training models through a customized road feature identification template, thereby avoiding the high cost of manual annotation, especially when different countries and regions require different road features, greatly reducing the cost of making data sets. At the same time, the disclosed method proposes to map a 3D point cloud to a 2D image through intensity projection, and extract road features using a road feature deep learning model corresponding to the 2D image, thereby solving the problem that traditional image methods lack spatial information and cannot obtain three-dimensional precise positioning. Subsequently, the disclosed method restores the information of the extracted 2D key points back to the 3D point cloud, thereby improving the extraction accuracy of road features. The disclosed method effectively overcomes the shortcomings of a single mode by fusing the texture information of the image and the spatial information of the point cloud, such as the problem that the image is greatly affected by light and weather and the point cloud data is sparse. The scheme improves the extraction accuracy and robustness through multimodal information. In addition, the disclosed method adopts a combination of the YOLOv8-OBB model and the PCA algorithm to quickly extract the rotation box of the road feature, and combines CenterNet for key point detection, thereby improving the real-time performance of the scheme in practical applications. In addition, the present disclosure combines the detection of the OBB rotation box with the PCA algorithm to perform principal component analysis on the rotation box of the road element and automatically rotate and align it, so that the extracted road elements can handle situations with large rotation angles, which is particularly suitable for complex road environments. Furthermore, users can customize or revise the road element identification template according to actual needs, such as element type, key points and other information, so that the road element deep learning model is highly adaptable and can flexibly handle a variety of road elements, suitable for different regions and scenarios.

[0073] In summary, the disclosed solution solves the problems in actual production, such as low road element extraction accuracy, poor robustness, difficulty in projecting into 3D point cloud data, high labor cost for data set production, and poor real-time performance. On this basis, the accuracy of the constructed map can be improved, which is conducive to improving the safety and convenience of autonomous driving.

[0074] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0076] Figure 1 is a flow chart of a method for extracting features based on zero-sample deep learning according to the present disclosure;

[0077] Figure 2Ais a schematic diagram of a two-dimensional intensity projection image according to an embodiment of the present disclosure;

[0078] Figure 2B is a schematic diagram of a road element identification template according to an embodiment of the present disclosure;

[0079] Figure 2C is a schematic diagram of a road element identification template including key points according to an embodiment of the present disclosure;

[0080] Figure 3 is a schematic diagram of a road element deep learning model according to an embodiment of the present disclosure;

[0081] Figure 4 is a structural schematic diagram of a feature extraction device based on zero-sample deep learning according to the present disclosure;

[0082] Figure 5 is a schematic structural diagram of an electronic device according to the present disclosure. DETAILED DESCRIPTION

[0083] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0084] In view of the problems of low road feature extraction accuracy, poor robustness, difficulty in projecting into 3D point cloud data, high labor cost for data set production, and poor real-time performance in actual production, the present invention proposes a feature extraction and map generation method and device based on zero-sample deep learning. The scheme of the present invention solves the above problems, and on this basis can improve the accuracy of the constructed map, which is conducive to improving the safety and speed of autonomous driving.

[0085] The technical solution of the present disclosure is described below through specific embodiments.

[0086] like Figure 1 As shown, it is a flowchart of the method for extracting features based on zero-sample deep learning of this embodiment. The execution subject of this embodiment is a computing device or component with data processing capability. The specific method of this embodiment may include the following steps:

[0087] S110, obtaining a pre-trained road element deep learning model and a three-dimensional point cloud image that requires road element extraction.

[0088] The road feature deep learning model here is trained using an unlabeled sample dataset.

[0089] S120 , performing intensity projection on the three-dimensional point cloud image to obtain a two-dimensional intensity projection image.

[0090] The three-dimensional information is projected onto a two-dimensional image, so that the subsequent road element deep learning model can extract key point information with three-dimensional features. Figure 2A As shown, a two-dimensional intensity projection image is obtained by intensity projection of a three-dimensional point cloud image.

[0091] S130. Input the two-dimensional intensity projection image into the road element deep learning model to obtain the two-dimensional coordinate information of the key points of the road elements output by the road element deep learning model; the road element deep learning model includes a YOLOv8-OBB rotating box model, a computing module including a PCA algorithm, and a multi-head centernet key point detection model in sequence.

[0092] The YOLOv8-OBB rotation box model is used to extract or determine the information of the rotation box of road elements. The PCA algorithm is used to rotate the pixel points corresponding to the extracted road elements. The multi-head centernet key point detection model is used to extract the two-dimensional information of key points based on the rotation of pixel points.

[0093] The dataset used to train the YOLOv8-OBB rotating box model and the dataset used to train the multi-head centernet key point detection model are generated by custom road element identification templates, which do not require annotation, thereby avoiding the manual annotation cost of dataset generation. The above custom road element identification template is a template that the user manually identifies or extracts on the two-dimensional intensity projection image according to the needs, and sets the key points of the template, such as Figure 2B The following is a schematic diagram of the road element identification template. Figure 2C Shown is a schematic diagram of key points on the road feature identification template.

[0094] In some embodiments, a road element deep learning model is constructed based on the category data of the road element identification template and the number of key points of the corresponding category. In this embodiment, the YOLOv8-OBB rotating box model and the multi-head centernet key point detection model are used together to construct a deep learning model for road element extraction.

[0095] S140. Determine the two-dimensional coordinate information of the key point on the two-dimensional intensity projection image according to the reverse logic of the processing logic of the road element deep learning model.

[0096] The two-dimensional information of the extracted key points is mapped onto a two-dimensional intensity projection image.

[0097] S150, determining the three-dimensional coordinates of the road element on the three-dimensional point cloud image according to the reverse logic of the processing logic of the intensity projection and the two-dimensional coordinate information of the key point on the two-dimensional intensity projection image.

[0098] The two-dimensional information of key points mapped to the two-dimensional intensity projection image is mapped to the three-dimensional point cloud image.

[0099] The data set used to train the multi-head centernet key point detection model is obtained according to the following steps:

[0100] Step 1: Acquire multiple three-dimensional point cloud sample images, and perform intensity projection on each three-dimensional point cloud sample image to obtain multiple two-dimensional intensity sample projection images.

[0101] Here, the algorithm for projecting the three-dimensional point cloud sample image into the two-dimensional intensity sample projection image is the same as the algorithm for performing intensity projection in S120.

[0102] Step 2: Using each two-dimensional intensity sample projection image, determine multiple road element identification templates and key points corresponding to each road element identification template.

[0103] Here, the road element identification template and the key points corresponding to each road element identification template can be manually determined from the two-dimensional intensity sample projection image.

[0104] Step three, fill each road element identification template with the first target scene color, and rotate them at least once according to the first preset angle range to obtain multiple first road element template images, and use the data set formed by the first road element template images as the data set for training the multi-head centernet key point detection model.

[0105] The first target scene color may be white, or other colors matching the scene.

[0106] The data set used to train the YOLOv8-OBB rotating box model is obtained according to the following steps:

[0107] Step 1: fill each road element identification template with the second target scene color, and rotate it at least once according to the second preset angle range to obtain multiple second road element template images; paste the obtained second road element template image onto at least one road background image according to at least one randomly determined position to obtain multiple training sample images.

[0108] The second target scene color may be white, or other colors matching the scene.

[0109] Although the above-mentioned pasting positions are random, they are known quantities and therefore do not require manual labeling.

[0110] Step 2: Use the data set formed by the training sample images as a data set for training the YOLOv8-OBB rotating box model.

[0111] The above data set may include a training set and a test set, and the training set and the test set are used together to complete the model training.

[0112] Among them, the road element deep learning model is as follows Figure 3 As shown, the processing logic of the road element deep learning model is as follows:

[0113] Step 1: Use the YOLOv8-OBB rotating box model to extract the rotating box position information of the road element from the two-dimensional intensity projection image; the rotating box position information is two-dimensional coordinate information.

[0114] The rotating box position information extracted by the above YOLOv8-OBB rotating box model can be four two-dimensional point coordinates. In addition, the YOLOv8-OBB rotating box model can also output the category information of the rotating box.

[0115] Step 2: The calculation module uses the PCA algorithm to perform principal component analysis on the rotation frame position information of the road element to obtain the main direction of the rotation frame.

[0116] Specifically, firstly, the two-dimensional coordinates of the centroid of the rotating frame are determined according to the rotating frame position information of the road element.

[0117] The two-dimensional coordinates of the center of mass of the rotating box can be determined using the following formula:

[0118]

[0119] Among them, (C x , C y ) represents the two-dimensional coordinates of the center of mass, (x i ,y i ) represents the coordinates of the boundary point i, and n represents the number of boundary points of the rotation box.

[0120] Afterwards, the two-dimensional coordinates of the centroid are used to perform decentralized processing on each boundary point of the rotation frame to obtain a decentralized point corresponding to each boundary point.

[0121] For each boundary point, the decentralization process is performed using the following formula:

[0122] P′ i =(x i -C x ,y i -C y )

[0123] Among them, (C x , C y ) represents the two-dimensional coordinates of the center of mass, (x i ,y i ) represents the coordinates of the boundary point i, P′ i Represents the decentralized point obtained by decentralized processing.

[0124] Afterwards, the covariance matrix is ​​determined using the coordinates of each decentralized point.

[0125] The covariance matrix Cov is as follows:

[0126]

[0127] Finally, the covariance matrix is ​​subjected to eigenvalue decomposition, and the main direction of the rotating frame is determined according to the result of the eigenvalue decomposition.

[0128] The covariance matrix is ​​decomposed by eigenvalues. The eigenvector gives the main axis direction of the OBB (OrientedboundingBox). The direction with a large eigenvalue is the long axis direction of the OBB, and the direction with a smaller eigenvalue is the short axis direction. Among them, the eigenvector V represents the direction of the OBB, and the eigenvalue Λ represents the variance of the data in this direction, as follows:

[0129] Cov=VΛV T

[0130] Step 3: According to the angle between the main direction of the rotation frame and the vertical direction, the pixel points corresponding to the road features on the two-dimensional intensity projection image are rotated.

[0131] Step 4: Use the multi-head centernet key point detection model to extract the key points of the rotated road elements and obtain the two-dimensional coordinate information of the key points.

[0132] In some embodiments, intensity projection is performed on the three-dimensional point cloud image to obtain a two-dimensional intensity projection image, which can be specifically achieved by the following steps:

[0133] Step 1: Determine the width and height of the point cloud according to the position information of each point in the three-dimensional point cloud image.

[0134] Specifically, a point in the point cloud is represented as:

[0135]

[0136] In the formula, (x i ,y i , Z i ) is the coordinate of point i, I iis the intensity information of point i, and m is the number of points in the 3D point cloud image.

[0137] Then, according to the position information of each point in the three-dimensional point cloud image, the maximum value max of the point cloud in the x direction is determined. x and minimum value min x ,Right now

[0138]

[0139] According to the position information of each point in the three-dimensional point cloud image, determine the maximum value max of the point cloud in the y direction y and minimum value min y ,Right now

[0140]

[0141] After that, the width of the point cloud is determined according to the maximum value of the point cloud in the x direction, the minimum value of the point cloud in the x direction, and the preset resolution, that is,

[0142]

[0143] In the formula, the preset resolution is set according to the actual situation, for example, it is set to 0.2.

[0144] According to the maximum value of the point cloud in the y direction, the minimum value of the point cloud in the y direction, and the preset resolution, the height of the point cloud is determined, that is,

[0145]

[0146] The x direction and the y direction are perpendicular to each other, the x direction may be a horizontal direction, and the y direction may be a vertical direction.

[0147] Step 2: Map each point in the three-dimensional point cloud image to the image grid according to the width and height of the point cloud, and obtain the position index of each point in the two-dimensional intensity projection image.

[0148] For each point, the first index of the point in the y direction is determined according to the coordinate of the point in the y direction, the maximum value of the point cloud in the y direction and the preset resolution; the second index of the point in the x direction is determined according to the coordinate of the point in the x direction, the maximum value of the point cloud in the x direction and the preset resolution; the first index and the second index are used as the position index of the point in the two-dimensional intensity projection image.

[0149] Specifically, the position index (A, B) of point i can be determined using the following formula:

[0150]

[0151] A and B are the indices of the columns and rows of the points in the two-dimensional intensity projection image.

[0152] This step realizes mapping the 3D point cloud into the image mesh.

[0153] Step 3: For an image grid with only one point, fill the intensity of the point in the image grid into the corresponding image network; for an image grid with multiple point clouds, fill the maximum intensity into the corresponding image network.

[0154] It can be expressed using the following formula:

[0155] grid intensity[A][B]=max(grid intensity[a][B],I i )

[0156] Grid intensity is the strength of the image network.

[0157] The above embodiments are proposed to address the problems of low extraction accuracy, poor robustness, difficulty in projecting into 3D point cloud data, high labor cost for data set production, and poor real-time performance when automatically extracting road elements when building high-precision maps, and can effectively solve the above problems.

[0158] The present disclosure provides a map generation method, comprising:

[0159] Step 1: Use the feature extraction method based on zero-sample deep learning in any of the above embodiments to determine the three-dimensional coordinates of each road feature in the three-dimensional point cloud image on the three-dimensional point cloud image.

[0160] Step 2: Generate a map including road elements based on the determined three-dimensional coordinates.

[0161] Using the zero-sample deep learning-based feature extraction method of any of the above embodiments to construct a map can improve the accuracy of the constructed map, thereby helping to improve the safety and speed of autonomous driving.

[0162] Based on the same inventive concept, the present disclosure provides a feature extraction device based on zero-sample deep learning. The steps performed by the components of the device are the same or similar to the above-mentioned feature extraction method based on zero-sample deep learning, so the similar parts are not repeated. Figure 4 As shown, the element extraction device based on zero-sample deep learning in this embodiment includes:

[0163] The data acquisition module 410 is used to obtain a pre-trained road element deep learning model and a three-dimensional point cloud image that requires road element extraction.

[0164] The intensity projection module 420 is used to perform intensity projection on the three-dimensional point cloud image to obtain a two-dimensional intensity projection image.

[0165] The key point extraction module 430 is used to input the two-dimensional intensity projection image into the road element deep learning model to obtain the two-dimensional coordinate information of the key points of the road elements output by the road element deep learning model; the road element deep learning model includes a YOLOv8-OBB rotating box model, a computing module including a PCA algorithm, and a multi-head centernet key point detection model in sequence.

[0166] The first reverse logic processing module 440 is used to determine the two-dimensional coordinate information of the key point two-dimensional coordinate information on the two-dimensional intensity projection image according to the reverse logic of the processing logic of the road element deep learning model.

[0167] The second reverse logic processing module 450 is used to determine the three-dimensional coordinates of the road element on the three-dimensional point cloud image according to the reverse logic of the intensity projection processing logic and the two-dimensional coordinate information of the key point on the two-dimensional intensity projection image.

[0168] The key point extraction module 430 is also used to determine a data set for training the multi-head centernet key point detection model:

[0169] Acquire multiple three-dimensional point cloud sample images, and perform intensity projection on each of the three-dimensional point cloud sample images to obtain multiple two-dimensional intensity sample projection images;

[0170] Determine a plurality of road element identification templates and key points corresponding to each road element identification template by using each two-dimensional intensity sample projection image;

[0171] Fill each road element identification template with the first target scene color, and rotate them at least once according to the first preset angle range to obtain multiple first road element template images, and use the data set formed by the first road element template images as the data set for training the multi-head centernet key point detection model;

[0172] The key point extraction module 430 is also used to determine a data set for training the YOLOv8-OBB rotating box model:

[0173] Filling each road element identification template with a second target scene color, and rotating each of the road element identification templates at least once according to a second preset angle range, to obtain a plurality of second road element template images; pasting the obtained second road element template images onto at least one road background image according to at least one randomly determined position, to obtain a plurality of training sample images;

[0174] Using the data set formed by the training sample images as a data set for training the YOLOv8-OBB rotating box model;

[0175] Among them, the processing logic of the road element deep learning model is as follows:

[0176] The YOLOv8-OBB rotating frame model is used to extract the rotating frame position information of the road element from the two-dimensional intensity projection image; the rotating frame position information is two-dimensional coordinate information;

[0177] Use the PCA algorithm to perform principal component analysis on the rotation frame position information of road elements to obtain the main direction of the rotation frame;

[0178] According to the angle between the main direction of the rotation frame and the vertical direction, the pixel points corresponding to the road elements on the two-dimensional intensity projection image are rotated;

[0179] The multi-head CenterNet key point detection model is used to extract the key points of the rotated road elements to obtain the two-dimensional coordinate information of the key points.

[0180] Based on the same inventive concept, the present disclosure provides a map generating device, the steps performed by the components of the device are the same or similar to those of the above-mentioned map generating method, so similar parts are not repeated. The map generating device of this embodiment includes:

[0181] The three-dimensional coordinate determination module is used to determine the three-dimensional coordinates of each road element in the three-dimensional point cloud image on the three-dimensional point cloud image by using the feature extraction method based on zero-sample deep learning of any of the above embodiments.

[0182] The map generation module is used to generate a map including road elements according to the determined three-dimensional coordinates.

[0183] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device and a computer-readable storage medium.

[0184] Figure 5 A schematic block diagram of an example electronic device 500 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0185] like Figure 5As shown, the device 500 includes a computing unit 510, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 520 or a computer program loaded from a storage unit 580 into a random access memory (RAM) 530. In the RAM 530, various programs and data required for the operation of the device 500 can also be stored. The computing unit 510, the ROM 520, and the RAM 530 are connected to each other via a bus 540. An input / output (I / O) interface 550 is also connected to the bus 540.

[0186] A number of components in the device 500 are connected to the I / O interface 550, including: an input unit 560, such as a keyboard, a mouse, etc.; an output unit 570, such as various types of displays, speakers, etc.; a storage unit 580, such as a disk, an optical disk, etc.; and a communication unit 590, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 590 allows the device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0187] The computing unit 510 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 510 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 510 performs the various methods and processes described above. For example, in some embodiments, any of the above methods may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 580. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 500 via ROM520 and / or communication unit 590. When the computer program is loaded into RAM530 and executed by the computing unit 510, one or more steps of any of the methods described above may be performed. Alternatively, in other embodiments, the computing unit 510 may be configured to perform any of the methods described above in any other appropriate manner (e.g., by means of firmware).

[0188] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0189] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0190] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0191] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0192] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0193] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0194] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0195] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for extracting features based on zero-sample deep learning, characterized in that: include: Obtain a pre-trained road feature deep learning model and a 3D point cloud image that requires road feature extraction; Performing intensity projection on the three-dimensional point cloud image to obtain a two-dimensional intensity projection image; Inputting the two-dimensional intensity projection image into the road element deep learning model to obtain the two-dimensional coordinate information of the key points of the road elements output by the road element deep learning model; the road element deep learning model includes a YOLOv8-OBB rotating box model, a computing module including a PCA algorithm, and a multi-head centernet key point detection model in sequence; Determine the two-dimensional coordinate information of the key point two-dimensional coordinate information on the two-dimensional intensity projection image according to the reverse logic of the processing logic of the road element deep learning model; Determine the three-dimensional coordinates of the road element on the three-dimensional point cloud image according to the reverse logic of the processing logic of the intensity projection and the two-dimensional coordinate information of the key point on the two-dimensional intensity projection image; The data set used to train the multi-head centernet key point detection model is obtained according to the following steps: Acquire multiple three-dimensional point cloud sample images, and perform intensity projection on each of the three-dimensional point cloud sample images to obtain multiple two-dimensional intensity sample projection images; Determine a plurality of road element identification templates and key points corresponding to each road element identification template by using each two-dimensional intensity sample projection image; Fill each road element identification template with the first target scene color, and rotate them at least once according to the first preset angle range to obtain multiple first road element template images, and use the data set formed by the first road element template images as the data set for training the multi-head centernet key point detection model; The data set used to train the YOLOv8-OBB rotating box model is obtained according to the following steps: Filling each road element identification template with a second target scene color, and rotating each of the road element identification templates at least once according to a second preset angle range, to obtain a plurality of second road element template images; pasting the obtained second road element template images onto at least one road background image according to at least one randomly determined position, to obtain a plurality of training sample images; Using the data set formed by the training sample images as a data set for training the YOLOv8-OBB rotating box model; Among them, the processing logic of the road element deep learning model is as follows: The YOLOv8-OBB rotating frame model is used to extract the rotating frame position information of the road element from the two-dimensional intensity projection image; the rotating frame position information is two-dimensional coordinate information; Use the PCA algorithm to perform principal component analysis on the rotation frame position information of road elements to obtain the main direction of the rotation frame; According to the angle between the main direction of the rotation frame and the vertical direction, the pixel points corresponding to the road elements on the two-dimensional intensity projection image are rotated; The multi-head CenterNet key point detection model is used to extract the key points of the rotated road elements to obtain the two-dimensional coordinate information of the key points.

2. The method according to claim 1, characterized in that The step of performing intensity projection on the three-dimensional point cloud image to obtain a two-dimensional intensity projection image includes: Determine the width and height of the point cloud according to the position information of each point in the three-dimensional point cloud image; Mapping each point in the three-dimensional point cloud image to an image grid according to the width and height of the point cloud to obtain a position index of each point in the two-dimensional intensity projection image; For an image grid with only one point, the intensity of the point in the image grid is filled into the corresponding image network; for an image grid with multiple point clouds, the maximum intensity is filled into the corresponding image network.

3. The method according to claim 2, characterized in that Determining the width and height of the point cloud according to the position information of each point in the three-dimensional point cloud image includes: Determine the maximum and minimum values ​​of the point cloud in the x direction according to the position information of each point in the three-dimensional point cloud image; Determine the width of the point cloud according to the maximum value of the point cloud in the x direction, the minimum value of the point cloud in the x direction, and the preset resolution; Determine the maximum and minimum values ​​of the point cloud in the y direction according to the position information of each point in the three-dimensional point cloud image; Determine the height of the point cloud according to the maximum value of the point cloud in the y direction, the minimum value of the point cloud in the y direction, and the preset resolution; The x direction and the y direction are perpendicular to each other.

4. The method according to claim 3, characterized in that Mapping each point in the three-dimensional point cloud image to an image grid according to the width and height of the point cloud to obtain a position index of each point in the two-dimensional intensity projection image includes: For each point, the first index of the point in the y direction is determined according to the coordinate of the point in the y direction, the maximum value of the point cloud in the y direction and the preset resolution; the second index of the point in the x direction is determined according to the coordinate of the point in the x direction, the maximum value of the point cloud in the x direction and the preset resolution; the first index and the second index are used as the position index of the point in the two-dimensional intensity projection image.

5. The method according to claim 1, characterized in that The PCA algorithm is used to perform principal component analysis on the rotation frame position information of the road element to obtain the main direction of the rotation frame, including: Determine the two-dimensional coordinates of the centroid of the rotating frame according to the rotating frame position information of the road element; Decentralizing each boundary point of the rotation frame using the two-dimensional coordinates of the centroid to obtain a decentralized point corresponding to each boundary point; Using the coordinates of each decentralized point, determine the covariance matrix; Perform eigenvalue decomposition on the covariance matrix, and determine the main direction of the rotating frame according to the result of the eigenvalue decomposition.

6. The method according to claim 5, characterized in that The decentralized processing of each boundary point of the rotation frame by using the two-dimensional coordinates of the centroid to obtain decentralized points corresponding to each boundary point includes: For each boundary point, the decentralization process is performed using the following formula: P′ i =(x i -C x ,y i -C y ) Among them, (C x , C y ) represents the two-dimensional coordinates of the center of mass, (x i ,y i ) represents the coordinates of the boundary point i, P′ i Represents the decentralized point obtained by decentralized processing.

7. The method according to claim 1, characterized in that The first target scene color and / or the second target scene color is white.

8. A feature extraction device based on zero-sample deep learning, characterized in that: include: A data acquisition module is used to obtain a pre-trained road element deep learning model and a three-dimensional point cloud image that needs to be extracted from the road element; An intensity projection module, used for performing intensity projection on the three-dimensional point cloud image to obtain a two-dimensional intensity projection image; A key point extraction module, used for inputting the two-dimensional intensity projection image into the road element deep learning model to obtain the two-dimensional coordinate information of the key points of the road elements output by the road element deep learning model; the road element deep learning model includes a YOLOv8-OBB rotating box model, a calculation module including a PCA algorithm, and a multi-head centernet key point detection model in sequence; A first reverse logic processing module, used to determine the two-dimensional coordinate information of the key point two-dimensional coordinate information on the two-dimensional intensity projection image according to the reverse logic of the processing logic of the road element deep learning model; A second reverse logic processing module, used for determining the three-dimensional coordinates of the road element on the three-dimensional point cloud image according to the reverse logic of the processing logic of the intensity projection and the two-dimensional coordinate information of the key point on the two-dimensional intensity projection image; Among them, the key point extraction module is also used to determine the data set used to train the multi-head centernet key point detection model: Acquire multiple three-dimensional point cloud sample images, and perform intensity projection on each of the three-dimensional point cloud sample images to obtain multiple two-dimensional intensity sample projection images; Determine a plurality of road element identification templates and key points corresponding to each road element identification template by using each two-dimensional intensity sample projection image; Fill each road element identification template with the first target scene color, and rotate them at least once according to the first preset angle range to obtain multiple first road element template images, and use the data set formed by the first road element template images as the data set for training the multi-head centernet key point detection model; Among them, the key point extraction module is also used to determine the data set for training the YOLOv8-OBB rotating box model: Filling each road element identification template with a second target scene color, and rotating each of the road element identification templates at least once according to a second preset angle range, to obtain a plurality of second road element template images; pasting the obtained second road element template images onto at least one road background image according to at least one randomly determined position, to obtain a plurality of training sample images; Using the data set formed by the training sample images as a data set for training the YOLOv8-OBB rotating box model; Among them, the processing logic of the road element deep learning model is as follows: The YOLOv8-OBB rotating frame model is used to extract the rotating frame position information of the road element from the two-dimensional intensity projection image; the rotating frame position information is two-dimensional coordinate information; Use the PCA algorithm to perform principal component analysis on the rotation frame position information of road elements to obtain the main direction of the rotation frame; According to the angle between the main direction of the rotation frame and the vertical direction, the pixel points corresponding to the road elements on the two-dimensional intensity projection image are rotated; The multi-head CenterNet key point detection model is used to extract the key points of the rotated road elements to obtain the two-dimensional coordinate information of the key points.

9. A map generation method, characterized in that: include: Determine the three-dimensional coordinates of each road element in the three-dimensional point cloud image on the three-dimensional point cloud image using the method described in any one of claims 1 to 8; A map including road elements is generated based on the determined three-dimensional coordinates.

10. A map generating device, characterized in that: include: A three-dimensional coordinate determination module, used to determine the three-dimensional coordinates of each road element in the three-dimensional point cloud image on the three-dimensional point cloud image using the method described in any one of claims 1 to 8; The map generation module is used to generate a map including road elements according to the determined three-dimensional coordinates.

Images