Map generation methods, training methods, devices, electronic devices, and storage media

Abstract

This disclosure discloses a map generation method, training method, apparatus, electronic device, and storage medium, relating to the field of artificial intelligence technology, particularly to the fields of autonomous driving, deep learning, and computer vision. The specific implementation scheme is as follows: generating a fused feature map based on image data and point cloud data; extracting the first point feature sequence from the fused feature map; processing the first point feature sequence and the fused feature map based on an attention strategy to obtain target element information; and generating a map based on the target element information.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and more particularly to the fields of autonomous driving, deep learning, and computer vision technology, and can be applied to dynamically generated maps. Specifically, it relates to a map generation method, training method, device, electronic device, and storage medium. Background Technology

[0002] High-definition maps, also known as autonomous driving maps or high-resolution maps, are a new map data paradigm for autonomous vehicles. High-definition maps can provide autonomous vehicles with accurate and comprehensive road feature data.

[0003] With the development of artificial intelligence technology, deep learning technology and computer vision technology are widely used to generate high-precision maps. Summary of the Invention

[0004] This disclosure provides a map generation method, a training method, an apparatus, an electronic device, and a storage medium.

[0005] According to one aspect of this disclosure, a map generation method is provided, comprising: generating a fused feature map based on image data and point cloud data; extracting a first point feature sequence from the fused feature map; processing the first point feature sequence and the fused feature map based on an attention strategy to obtain target element information; and generating a map based on the target element information.

[0006] According to another aspect of this disclosure, a method for training a deep learning model is provided, comprising: obtaining a first sample point feature sequence and a second sample point feature sequence based on a sample image, wherein the first sample point feature sequence includes S first sample points, the second sample point feature sequence includes S second sample points, S is an integer greater than 1, the sample image represents a fusion feature map generated based on image data and point cloud data, and the s-th second sample point of the second sample point feature sequence is the sample label of the s-th first sample point in the first sample point feature sequence, s is an integer greater than or equal to 1 and less than or equal to S; and training a deep learning model using the first sample point feature sequence and the second sample point feature sequence to obtain a trained deep learning model.

[0007] According to another aspect of this disclosure, a map generation apparatus is provided, comprising: a first generation module, an extraction module, a first acquisition module, and a second generation module. The first generation module is used to generate a fused feature map based on image data and point cloud data. The extraction module is used to extract a first point feature sequence from the fused feature map. The first acquisition module is used to process the first point feature sequence and the fused feature map based on an attention strategy to obtain target feature information. The second generation module is used to generate a map based on the target feature information.

[0008] According to another aspect of this disclosure, a training apparatus for a deep learning model is provided, comprising: a second obtaining module and a third obtaining module. The second obtaining module is configured to obtain a first sample point feature sequence and a second sample point feature sequence based on a sample image, wherein the first sample point feature sequence includes S first sample points, and the second sample point feature sequence includes S second sample points, where S is an integer greater than 1; the sample image represents a fused feature map generated from image data and point cloud data; and the s-th second sample point of the second sample point feature sequence is the sample label of the s-th first sample point in the first sample point feature sequence, where s is an integer greater than or equal to 1 and less than or equal to S. The third obtaining module is configured to train a deep learning model using the first sample point feature sequence and the second sample point feature sequence to obtain a trained deep learning model.

[0009] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method as described above.

[0010] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform the method described above.

[0011] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the method described above.

[0012] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0013] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0014] Figure 1 This illustration schematically shows an exemplary system architecture to which map generation methods and apparatus can be applied according to embodiments of the present disclosure;

[0015] Figure 2 A flowchart illustrating a map generation method according to an embodiment of the present disclosure is shown schematically;

[0016] Figure 3 The illustration schematically shows a principle diagram of generating a fused feature map according to an embodiment of the present disclosure;

[0017] Figure 4 The illustration schematically shows a principle diagram for generating a first point feature sequence according to an embodiment of the present disclosure;

[0018] Figure 5 A flowchart illustrating the extraction of a first sparse point sequence from a first discrete point sequence according to an embodiment of the present disclosure is shown.

[0019] Figure 6 A flowchart illustrating the extraction of a target discrete point sequence from a first sparse point sequence according to an embodiment of the present disclosure is shown.

[0020] Figure 7 This schematically illustrates a flowchart of a process for processing a first-point feature sequence and a fused feature map using an attention-based strategy according to an embodiment of the present disclosure to obtain target element information.

[0021] Figure 8 This schematically illustrates the principle of processing the first point feature sequence and the fused feature map according to an embodiment of the present disclosure based on an attention strategy to obtain the target feature information of the M+nth point;

[0022] Figure 9 The schematic diagram illustrates the principle of a map generation method according to an embodiment of the present disclosure;

[0023] Figure 10 A flowchart illustrating a training method for a deep learning model according to an embodiment of the present disclosure is shown schematically.

[0024] Figure 11 An exemplary architecture diagram of a deep learning model according to an embodiment of the present disclosure is illustrated schematically;

[0025] Figure 12 The illustration shows a schematic diagram of the principle of a training method for a deep learning model according to an embodiment of the present disclosure;

[0026] Figure 13 A block diagram of a map generation apparatus according to an embodiment of the present disclosure is shown schematically;

[0027] Figure 14 A block diagram of a deep learning model training apparatus according to an embodiment of the present disclosure is schematically shown; and

[0028] Figure 15 A block diagram of an electronic device suitable for implementing a map generation method and a deep learning model training method according to embodiments of the present disclosure is shown schematically. Detailed Implementation

[0029] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0030] High-definition maps can provide road topology information for autonomous vehicles. The construction of high-definition maps typically employs an offline method, involving the acquisition of images including lane lines, the extraction of lane lines from the images using image processing techniques, the conversion of the extracted lane lines to a 2D plane, and then the conversion to a high-definition map format.

[0031] However, converting image data into a 2D plane results in a loss of accuracy, leading to suboptimal precision in high-definition maps. Furthermore, in scenarios with complex road conditions and high timeliness requirements, this offline map generation method suffers from poor timeliness.

[0032] Therefore, this disclosure provides a map generation scheme. For example: generating a fused feature map based on image data and point cloud data; extracting a first point feature sequence from the fused feature map; processing the first point feature sequence and the fused feature map based on an attention strategy to obtain target element information; and generating a map based on the target element information. This improves the accuracy and timeliness of map generation.

[0033] Figure 1 The illustration schematically shows an exemplary system architecture of a training method and apparatus for map generation or deep learning models that can be applied according to embodiments of the present disclosure.

[0034] It is important to note that Figure 1The examples shown are merely examples of system architectures applicable to embodiments of this disclosure, intended to help those skilled in the art understand the technical content of this disclosure. They do not imply that embodiments of this disclosure cannot be used in other devices, systems, environments, or scenarios. For instance, in another embodiment, an exemplary system architecture for applying map generation or deep learning model training methods and apparatus may include a terminal device. However, the terminal device can implement the map generation or deep learning model training methods and apparatus provided in the embodiments of this disclosure without interacting with a server.

[0035] like Figure 1 As shown, the system architecture 100 according to this embodiment may include a vehicle terminal device 101, a network 102, and a server 103. The network 102 serves as a medium for providing a communication link between the vehicle terminal device 101 and the server 103. The network 102 may include various connection types, such as wireless communication links, etc.

[0036] Users can use vehicle terminal device 101 to interact with server 103 via network 102 to receive or send messages, etc. Various communication client applications can be installed on vehicle terminal device 101, such as knowledge reading applications, web browser applications, search applications, instant messaging tools, email clients and / or social platform software, etc. (for example only).

[0037] The vehicle terminal device 101 can be various in-vehicle electronic devices with a display screen and support for web browsing, including but not limited to smartphones, tablets, laptops, etc.

[0038] Server 103 can be a server that provides various services, such as a backend management server that supports the content browsed by the user using the vehicle terminal device (for example only). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0039] It should be noted that the map generation or deep learning model training method provided in this disclosure embodiment can generally be executed by a vehicle terminal device. Accordingly, the map generation or deep learning model training device provided in this disclosure embodiment can also be located in the vehicle terminal device 101.

[0040] Alternatively, the map generation or deep learning model training method provided in this embodiment can generally be executed by server 103. Correspondingly, the map generation or deep learning model training apparatus provided in this embodiment can generally be located in server 103. The map generation or deep learning model training method provided in this embodiment can also be executed by a server or server cluster that is different from server 103 and capable of communicating with vehicle terminal device 101 and / or server 103. Correspondingly, the map generation or deep learning model training apparatus provided in this embodiment can also be located in a server or server cluster that is different from server 103 and capable of communicating with vehicle terminal device 101 and / or server 103.

[0041] For example, while a user's vehicle is in motion, the vehicle terminal device 101 can acquire image data and point cloud data of the surrounding area through onboard acquisition devices. This acquired image data and point cloud data are then sent to the server 103, which generates a fused feature map based on the image data and point cloud data. The server then extracts the first point feature sequence from the fused map and processes the first point feature sequence and the fused feature map using an attention strategy to obtain target element information. Finally, a map is generated based on the target element information and sent to the vehicle terminal device 101. Alternatively, a server or server cluster capable of communicating with the vehicle terminal device 101 and / or the server 103 can process the image data and point cloud data to ultimately generate a map of the vicinity of the target vehicle.

[0042] It should be understood that Figure 1 The number of vehicle terminal devices, networks, and servers shown is merely illustrative. Any number of vehicle terminal devices, networks, and servers can be included depending on implementation needs.

[0043] Figure 2 A flowchart illustrating a method for training a map generation or deep learning model according to an embodiment of the present disclosure is shown.

[0044] like Figure 2 As shown, the method includes operations S210 to S240.

[0045] In operation S210, a fused feature map is generated based on the image data and point cloud data.

[0046] In operation S220, the first point feature sequence of the fused feature map is extracted.

[0047] In operation S230, based on the attention strategy, the first feature sequence and the fused feature map are processed to obtain the target element information.

[0048] In operation S240, a map is generated based on the target feature information.

[0049] According to embodiments of this disclosure, the image data can be obtained by capturing images of a target area using a vision sensor mounted on an autonomous vehicle. The target area can be the area surrounding the autonomous vehicle during its operation. The vision sensor can be a camera or a video camera. The image data can be in any format, such as RGB (Red, Green, Blue) format or YUV (Luminance, Chrominance) format.

[0050] According to embodiments of this disclosure, point cloud data can be obtained by scanning a target area using a lidar installed on an autonomous vehicle.

[0051] According to embodiments of this disclosure, image data can be used to provide road information around the vehicle body of an autonomous vehicle, such as lane line information, obstacle information, basic traffic infrastructure information, etc. Although point cloud data lacks the texture information found in image data, it can be used to provide 3D geometric information of roads at a distance from the autonomous vehicle body to supplement the information in the image data.

[0052] According to embodiments of this disclosure, image data and point cloud data can be fused to generate a fused feature map through various feature fusion methods. For example, image data can first be processed using deep learning methods based on images to obtain image features, such as convolutional neural networks. Then, point cloud data can be processed using deep learning methods based on point clouds to obtain point cloud features. For example, point cloud data can be arranged into a three-dimensional mesh at a fixed resolution; or point cloud data can be divided into imbalanced trees and processed according to the point density of regions. Finally, the image features and point cloud features are fused to obtain a fused feature map.

[0053] According to embodiments of this disclosure, in the fused feature map, each feature can include both location features and attribute features of the map element. Location features can characterize the pixel coordinate features of the map element, such as two-dimensional coordinate type features and coordinate values. Attribute features can characterize the category features of the map element, such as lane line attributes, obstacle attributes, etc.

[0054] According to embodiments of this disclosure, the first point feature sequence of the fused feature map can be extracted by methods such as object detection and semantic segmentation to extract the location features and attribute features of map elements in the fused feature map. Map elements represent feature points of the fused feature map, and the first point feature sequence can include the location features and attribute features of the feature points. For example, if the pixel coordinates of a feature point on a lane line are (2, 3), it can be represented that the location feature of the feature point is the x-type coordinate value of 2 and the y-type coordinate value of 3, and the attribute feature is the lane line attribute. The point features of this feature point can be represented as (x, 2, lane line) and (y, 3, lane line).

[0055] According to embodiments of this disclosure, the first point feature sequence extracted through object detection or semantic segmentation provides relatively concise map element information. A trained deep learning model, based on an attention strategy, can be used to process the first point feature sequence and the fused feature map to complete the map element information and obtain more comprehensive target element information.

[0056] According to embodiments of this disclosure, since each target feature information includes the location and attribute features of map features, a data association algorithm can be used to associate target features with each other based on their location and attribute features to obtain a map. The data association algorithm can be ICP (Iterative Closest Point), PPICP (Point to Plane ICP), GICP (Generalized ICP), or VGICP (Voxelized Generalized ICP), etc.

[0057] According to embodiments of this disclosure, operations S210 to S240 can be performed by an electronic device. The electronic device may include a server or a terminal device. The server may be... Figure 1 Server 103 in the middle. Vehicle terminal equipment can be Figure 1 Vehicle terminal equipment 101.

[0058] According to embodiments of this disclosure, a fused feature map is generated by combining image data and point cloud data. This allows for the fusion of feature features identified by a visual sensor and a LiDAR sensor, resulting in a fused feature map with high accuracy of map features. A first-point feature sequence is then extracted from the fused feature map. Based on an attention strategy, the first-point feature sequence and the fused feature map are processed to obtain information about the target feature. This process supplements the feature information of the simplified first-point feature sequence, improving the completeness of map feature features. Finally, a map is generated using the complete target feature information. Therefore, a map of the distance between an autonomous vehicle and its target area can be dynamically generated in real time without relying on offline maps. This is suitable for generating time-sensitive features in maps, improving map generation efficiency and accuracy.

[0059] The following is for reference. Figures 3-9 In conjunction with specific embodiments, Figure 2 The method shown will be further explained.

[0060] Figure 3 The illustration schematically shows a principle diagram of generating a fused feature map according to an embodiment of the present disclosure.

[0061] like Figure 3 As shown, in step 300, an image feature map 314 is generated based on a predetermined feature map 311, image features 312 of the image data, and a first parameter 316. A point cloud feature map 318 is generated based on a predetermined feature map 315, point cloud features 316 of the point cloud data, and a second parameter 317. The image feature map 314 and the point cloud feature map 318 are then fused to obtain a fused feature map 319.

[0062] According to embodiments of this disclosure, image features 312 of image data can characterize the location and attribute features of map elements in the image data. Image features 312 can be obtained by processing the image data using image feature extraction methods. Image feature extraction methods can include: Fourier transform, windowed Fourier transform, wavelet transform, least squares method, boundary orientation histogram method, etc. The above image feature extraction methods can be implemented using convolutional neural networks, such as Efficient-Net, a convolutional neural network that can implement model scaling.

[0063] According to embodiments of this disclosure, the predetermined feature map 311 and predetermined feature map 315 may be BEV feature maps (Bird's Eye View Feature Maps) that only follow a Gaussian distribution and do not contain any features.

[0064] According to embodiments of this disclosure, the first parameter 316 characterizes the intrinsic and extrinsic parameters of the vision sensor. The intrinsic parameters of the vision sensor characterize parameters related to the sensor's own characteristics, such as focal length and pixel count. The extrinsic parameters of the vision sensor characterize parameters related to the acquisition environment, such as the sensor's mounting position and rotation direction.

[0065] According to embodiments of this disclosure, image feature map 314 represents the BEV feature map of image features 312 containing image data. Image feature map 314 can be obtained by learning image features 312 of image data using a predetermined feature map 311 based on deep learning technology.

[0066] According to embodiments of this disclosure, point cloud features 316 of point cloud data can characterize the location features of the point cloud data. Point cloud features 316 are obtained by processing the point cloud data using a point cloud feature extraction network. For example, the point cloud feature extraction network can be a Point Pillar network.

[0067] According to embodiments of this disclosure, the second parameter 317 characterizes the intrinsic and extrinsic parameters of the lidar. The intrinsic parameters of the lidar characterize parameters related to the lidar's own characteristics, such as measurement distance, scanning frequency, and measurement resolution. The extrinsic parameters of the lidar characterize parameters related to the acquisition environment, such as the lidar's installation position and rotation direction.

[0068] According to embodiments of this disclosure, the point cloud feature map 318 represents the BEV feature map of the point cloud features 316 containing point cloud data. The point cloud feature map 318 can be obtained by learning the point cloud features 316 of the point cloud data using a predetermined feature map 315 based on deep learning technology.

[0069] According to embodiments of this disclosure, image feature map 314 and point cloud feature map 318 can be fused by feature splicing or feature overlay to obtain fused feature map 319.

[0070] According to embodiments of this disclosure, a fused feature map containing only random features following a Gaussian distribution is obtained by learning image features from image data and point cloud features from point cloud data using deep learning technology. This achieves the goal of obtaining the location and attribute features of map elements without requiring a conversion step from image data to a 2D planar viewpoint, thus improving the accuracy of map generation.

[0071] According to embodiments of this disclosure, operation S220 can extract the first point feature sequence of the fused feature map by means of target detection, using bounding box or SME (Start-Middle-End, interpolation sampling).

[0072] For example, the target feature of the fused feature map extracted using bounding boxes can be map feature A, and the category of map feature A can be lane lines. The bounding box surrounding map feature A can be a rectangle, and the coordinates of the four vertices of the rectangle can be (x1, y1), (x2, y2), (x3, y3), and (x4, y4). The first point feature sequence can be represented as [(x, x1, lane line), (y, y1, lane line), (x, x2, lane line), (y, y2, lane line), (x, x3, lane line), (y, y3, lane line), (x, x4, lane line), (y, y4, lane line)].

[0073] According to embodiments of this disclosure, operation S220 can also extract the first point feature sequence of the fused feature map through semantic segmentation. This includes the following operations:

[0074] The fused feature map is subjected to semantic segmentation to obtain a semantic feature map; the semantic feature map is subjected to instance segmentation to obtain an instance feature map; and the first feature sequence is extracted from the instance feature map.

[0075] According to embodiments of this disclosure, before performing semantic segmentation processing on the fused feature map, the fused feature map can be converted into a feature map that can be used for semantic segmentation processing by interpolation, point filling, or dilation.

[0076] According to embodiments of this disclosure, an image semantic segmentation model can be used to perform semantic segmentation processing on the fused feature map, such as a fully convolutional neural network model (FCN). A real-time instance segmentation model can be used to perform instance segmentation on the semantic feature map, such as a fully convolutional real-time instance segmentation model (SparseInst).

[0077] According to embodiments of this disclosure, extracting a first point feature sequence from an instance feature map may include the following operations:

[0078] The instance feature map is processed to obtain a discrete point set, which includes multiple discrete points and their feature information. Based on the feature information of the multiple discrete points, a target discrete point sequence is extracted from the multiple discrete points. The target discrete point sequence includes multiple target discrete points and their feature information. Finally, based on the feature information of the multiple target discrete points, a first point feature sequence is generated.

[0079] According to embodiments of this disclosure, processing the instance feature map to obtain a discrete point set may include the following operations:

[0080] The instance feature map is eroded to obtain a sparse instance feature map; and the sparse instance feature map is pooled to obtain a discrete point set.

[0081] According to embodiments of this disclosure, image processing algorithms can be used to perform sparse processing on the pixel features in the instance feature map to obtain a sparse instance feature map. In the sparse instance feature map, the pixel width of each map element can reach a predetermined threshold, for example, the predetermined threshold for the pixel width of a map element belonging to a lane line can be 5.

[0082] According to embodiments of this disclosure, pooling processing of sparse instance feature maps may include any one or both of max pooling and average pooling operations to obtain a discrete point set.

[0083] According to embodiments of this disclosure, extracting a target discrete point sequence from multiple discrete points based on feature information of multiple discrete points includes the following operations:

[0084] Based on the feature information of multiple discrete points, a discrete point sorting strategy is determined. Following this strategy, the multiple discrete points are sorted in a forward order to obtain a first discrete point sequence. Based on a selection strategy, a target discrete point sequence is extracted from this first discrete point sequence.

[0085] According to embodiments of this disclosure, the feature information of multiple discrete points characterizes the coordinate fluctuation range of the multiple discrete points. For example, for all discrete points, the fluctuation range of the horizontal coordinate can be 1 to 200, and the fluctuation range of the vertical coordinate can be 0.5 to 100. The determined sorting strategy for the discrete points can be to sort them in ascending order of their horizontal coordinates. Then, based on a selection strategy, a portion of the discrete points are extracted from the first discrete point sequence as the target discrete point sequence.

[0086] According to embodiments of this disclosure, extracting a portion of discrete points from a first discrete point sequence as a target discrete point sequence based on a selection strategy may include the following operations:

[0087] The first discrete point sequence is subjected to sparsification to obtain a first sparse point sequence. Following a sorting strategy, the first sparse point sequence is reverse-sorted to obtain a second sparse point sequence. Finally, the second sparse point sequence is subjected to sparsification to obtain the target discrete point sequence.

[0088] Figure 4 The illustration shows a schematic diagram of the principle of generating a first point feature sequence according to an embodiment of the present disclosure.

[0089] like Figure 4As shown, in step 400, semantic segmentation is performed on the fused feature map 421 to obtain a semantic feature map 422. Instance segmentation is then performed on the semantic feature map 422 to obtain an instance feature map 423. A discrete point set 424 is extracted from the instance feature map 423. Based on a ranking strategy, the discrete point set 424 is processed to obtain a first discrete point sequence 425. Based on a selection strategy, the first discrete point sequence 425 is processed to obtain a first sparse point sequence 426. Based on the aforementioned ranking strategy, the first sparse point sequence 426 is processed to obtain a second sparse point sequence 427. Based on the aforementioned selection strategy, the second sparse point sequence 427 is processed to obtain a target discrete point sequence 428. Based on the target discrete point sequence 428, a first point feature sequence 429 is obtained.

[0090] According to embodiments of this disclosure, the first discrete point sequence includes I discrete points, where I is an integer greater than 1. The following is combined with... Figure 5 and Figure 6 The method for extracting the target discrete point sequence from the first discrete point sequence based on the selection strategy is further explained.

[0091] Figure 5 A flowchart illustrating the extraction of a first sparse point sequence from a first discrete point sequence according to an embodiment of the present disclosure is shown.

[0092] like Figure 5 As shown, the method 500 includes operations S5510 to S5540.

[0093] In operation S5510, for the i-th discrete point among I discrete points, k discrete points are deleted to obtain the second discrete point sequence.

[0094] In operation S5520, the j-th discrete point is extracted from the second discrete point sequence as a sparse point, and i is incremented.

[0095] In operation S5530, determine whether i is less than I. If yes, then execute operation S5540. If not, return to execute operation S5510.

[0096] By operating S5540, the first sparse point sequence is obtained.

[0097] According to an embodiment of this disclosure, the first discrete point sequence is processed in I rounds to obtain a first sparse point sequence.

[0098] According to embodiments of this disclosure, the pixel distance between each of the k discrete points and the i-th discrete point satisfies a preset threshold range. For example, the preset threshold range may include two types: a distance of 3 to 5 pixels and a distance of more than 10 pixels.

[0099] For example, the first discrete point sequence may include discrete point A1, discrete point A2, ..., discrete point A i ... discrete point A I In the first round of processing of the first discrete point sequence, for the first discrete point A1, discrete points that satisfy a preset threshold range between it and A1 are deleted, resulting in the second discrete point sequence: discrete points A1, A2, ..., At. From the second discrete point sequence, the discrete point A1 with the closest pixel distance to discrete point A1 can be selected. t As the first sparse point in the first sparse point sequence, we can then increment i sequentially until i equals 1, thus obtaining the first sparse point sequence. For example, the first sparse point sequence may include Q discrete points.

[0100] According to embodiments of this disclosure, operations S5510 to S5540 can be performed by an electronic device. The electronic device may include a server or a terminal device. The server may be... Figure 1 Server 103 in the middle. Vehicle terminal equipment can be Figure 1 Vehicle terminal equipment 101.

[0101] Figure 6 A flowchart illustrating the extraction of a target discrete point sequence from a first sparse point sequence according to an embodiment of the present disclosure is shown.

[0102] like Figure 6 As shown, the method includes operations S6610 to S6650.

[0103] In operation S6610, the first sparse point sequence is reversed according to the sorting strategy to obtain the second sparse point sequence.

[0104] In operation S6620, for the qth discrete point out of Q discrete points, h discrete points are deleted to obtain the third sparse point sequence.

[0105] In operation S6630, the p-th discrete point is extracted from the third sparse point sequence as a sparse point, and q is incremented.

[0106] In operation S6640, determine whether q is less than Q. If yes, execute operation S6650. If not, execute operation S6610.

[0107] By operating the S6650, the target discrete point sequence is obtained.

[0108] According to embodiments of this disclosure, the sorting strategy can be to sort the discrete points in ascending order of their x-coordinates. Reversing the sorting strategy for the first sparse point sequence can be represented by sorting the first sparse point sequence in descending order of the x-coordinates of Q discrete points to obtain the second sparse point sequence.

[0109] According to embodiments of this disclosure, the pixel distance between each of the h discrete points and the qth discrete point satisfies a preset threshold range. For example, the preset threshold range may include two: a distance of 3 to 5 pixels and a distance of more than 10 pixels.

[0110] According to embodiments of this disclosure, the method for sparse processing of the second sparse point sequence is the same as the method for sparse processing of the first discrete point sequence to obtain the first sparse point sequence, and will not be described in detail here.

[0111] According to embodiments of this disclosure, the first sparse point sequence is reverse-sorted according to a sorting strategy, and then sparse processing is performed to obtain the target discrete point sequence, so as to improve the accuracy of the relative positions between discrete points located at adjacent sorting positions in the target discrete point sequence.

[0112] According to embodiments of this disclosure, operations S6610 to S6650 can be performed by an electronic device. The electronic device may include a server or a terminal device. The server may be... Figure 1 Server 103 in the middle. Vehicle terminal equipment can be Figure 1 Vehicle terminal equipment 101.

[0113] Figure 7 The flowchart illustrates a process of processing a first point feature sequence and a fused feature map using an attention-based strategy according to an embodiment of the present disclosure to obtain target element information.

[0114] like Figure 7 As shown, the method 700 includes operations S7310 to S7340.

[0115] When operating the S7310, based on the attention strategy, the first point feature sequence and the fused feature map are processed to obtain the target feature information of the M+n point.

[0116] In operation S7320, based on the target feature information of the (M+n)th point and the feature sequence of the first point, the feature sequence of the second point is generated and incremented by n.

[0117] In operation S7330, determine if i is less than N. If yes, proceed to operation S7340. If not, return to operation S7310.

[0118] When operating the S7340, target feature information is generated based on the target feature information of M+N points.

[0119] According to embodiments of this disclosure, an attention strategy can be used to focus on important information with high weight and ignore unimportant information with low weight, and can exchange information with other information by sharing important information, thereby realizing the transmission of important information. In embodiments of this disclosure, the attention strategy can extract information from the first point feature sequence itself, the interior of the fused feature map, and the information between the first point feature sequence and the fused feature map, so as to obtain target feature information of (M+n) points based on the feature information of M points in the first point feature sequence and the feature information in the fused feature map.

[0120] According to embodiments of this disclosure, the target feature information at the (M+n)th point is obtained by processing the feature sequence of the first point and the fused feature map based on an attention strategy. Therefore, the target feature information at the (M+n)th point participates in the global attention mechanism, coupling with the global information of the fused feature map, thereby improving the accuracy of the target feature information.

[0121] According to embodiments of this disclosure, the attention strategy may include a self-attention mechanism and an interactive attention mechanism. Operation S7310 may include the following operations:

[0122] Based on a self-attention mechanism, the first feature sequence is processed to obtain the first feature information of the (M+n)th point. Based on the first feature sequence and the first feature information of the (M+n)th point, a third feature sequence is generated. Based on an interactive attention mechanism, the third feature sequence and the fused feature map are processed to obtain the target feature information of the (M+n)th feature point.

[0123] According to embodiments of this disclosure, the first feature information of the (M+n)th point is obtained by processing the feature sequence of the first point based on a self-attention mechanism. Therefore, the first feature information of the (M+n)th point is obtained based on the interaction of the positional and attribute features of each feature point in the feature sequence with other feature points.

[0124] For example, the first feature sequence can include the first feature sequence of M points. The coordinate arrangement of the M points shows a certain trend. Based on the self-attention mechanism, the processing of the first feature sequence can predict the position features and attribute features of the M+nth point according to the trend of the coordinate arrangement.

[0125] According to embodiments of this disclosure, the third feature sequence includes both the initial first feature sequence and the first feature information of the predicted point (the (M+n)th point). Based on the interactive attention mechanism, the target feature information of the predicted point is obtained by interacting with the feature information of the feature points in the third feature sequence and the feature information of the feature points in the fused feature map. This couples the feature information of the fused feature map and improves the accuracy of the target feature information of the predicted point.

[0126] According to embodiments of this disclosure, operations S7310 to S7340 can be performed by an electronic device. The electronic device may include a server or a terminal device. The server may be... Figure 1 Server 103 in the middle. Vehicle terminal equipment can be Figure 1 Vehicle terminal equipment 101.

[0127] Figure 8 The illustration shows a schematic diagram of the principle of processing the first point feature sequence and the fused feature map according to an embodiment of the present disclosure based on an attention strategy to obtain the target feature information of the M+nth point.

[0128] like Figure 8 As shown, in 800, the first feature sequence 429 includes feature information of M points, namely the feature information of the 1st point 429_1, ..., the feature information of the Mth point 429_M. After n rounds of processing, the target feature information of the (M+n)th point is obtained.

[0129] For example, in the first round of processing, the feature information of the M points in the first feature sequence 429 is processed based on the self-attention mechanism to obtain the first feature information 801_1 of the (M+1)th point. Based on the first feature sequence 429 and the first feature information 801_1 of the (M+1)th point, the third feature sequence 802 is obtained. The third feature sequence 802 includes the feature information of the original M points (802_1, ..., 802_M) in the first feature sequence and the newly generated first feature information 802_M+1 of the (M+1)th point. Then, based on the interactive attention mechanism, the third feature sequence 802 and the fused feature map 319 are processed to obtain the target feature information of the (M+1)th point.

[0130] In the second round of processing, the target feature information of the (M+1)th point and the first feature sequence 429 are first processed based on the self-attention mechanism to obtain the first feature information of the (M+2)th point. Then, the third feature sequence 802 is generated based on the first feature sequence 429, the target feature information of the (M+1)th point, and the first feature information of the (M+2)th point. At this point, the third feature sequence 802 includes the feature information of the original M points in the first feature sequence, the target feature information of the (M+1)th point, and the first feature information of the (M+2)th point obtained in the first round of processing. Then, based on the interactive attention mechanism, the third feature sequence 429 and the fused feature map 319 are processed to obtain the target feature information of the (M+2)th point. This process is repeated until N rounds of processing are performed, resulting in the target feature information of M+N points.

[0131] Figure 9 The schematic diagram illustrates the principle of a map generation method according to an embodiment of the present disclosure.

[0132] like Figure 9 As shown, in step 900, the original image 901 undergoes feature extraction to obtain image features 902, and the predetermined feature map is learned from the image features 902 to obtain a fused feature map. The original image 901 may include an RGB image acquired by a visual sensor and a point cloud image acquired by a LiDAR. The image features 902 may include features from the RGB image and features from the point cloud image.

[0133] The fused feature map 903 is processed by object detection or semantic segmentation to obtain a discrete point set 904. In the embodiments of this disclosure, the discrete point set obtained by object detection does not require sparsity processing to obtain the first point feature sequence 905. The discrete point set obtained by semantic segmentation requires sparsity processing to obtain the first point feature sequence 905.

[0134] The first feature sequence 905 is input into the trained deep learning model, which outputs the target element 906 and generates a map 907 based on the target map element 906.

[0135] For example: the first feature sequence 905(x1, y1, x2, y2, ..., x n y n In a trained deep learning model, after the first round of processing, the resulting point feature sequence can be (x1, y1, x2, y2, ..., x...). n y n (x^1, y^1). After a second round of processing, the resulting point feature sequence can be (x1, y1, x2, y2, ..., x...). n y n (x^1, y^1, x^2, y^2). Continue until the deep learning model outputs a stop symbol, at which point n = N, and the target element information (x^1, y^1, x^2, y^2, ..., x^2) is obtained. N y^ N ).

[0136] It should be noted that the feature points in the first feature sequence include not only location features but also attribute features. Since the attribute features of the first feature sequence obtained for the same instance feature map are generally the same during instance segmentation, attribute features are not labeled in the above example.

[0137] Figure 10 A flowchart illustrating a training method for a deep learning model according to an embodiment of the present disclosure is shown.

[0138] like Figure 10 As shown, the method 1000 includes operations S1010 to S1020.

[0139] In operation S1010, the first sample point feature sequence and the second sample point feature sequence are obtained based on the sample image.

[0140] In operation S1020, the deep learning model is trained using the feature sequences of the first and second sample points to obtain the trained deep learning model.

[0141] According to an embodiment of this disclosure, the first sample point feature sequence includes S first sample points, the second sample point feature sequence includes S second sample points, where S is an integer greater than 1, the sample image represents a fusion feature map generated based on image data and point cloud data, and the s-th second sample point in the second sample point feature sequence is the sample label of the s-th first sample point in the first sample point feature sequence, where s is an integer greater than or equal to 1 and less than or equal to S.

[0142] According to embodiments of this disclosure, operation S1020 may include the following operations:

[0143] Based on an attention strategy, the feature sequences of the first and second sample points are processed to obtain the feature sequence of the third sample points, which includes S third sample points. Based on a loss function, the s-th loss value is generated according to the s-th third sample point in the feature sequence and the s-th second sample point in the feature sequence. The model parameters of the deep learning model are adjusted based on the loss values ​​corresponding to the S third sample points to obtain the trained deep learning model.

[0144] Figure 11 An exemplary architecture diagram of a deep learning model according to an embodiment of the present disclosure is illustrated.

[0145] like Figure 11 As shown, the deep learning model 1100 includes a processing module 1101 based on a self-attention mechanism, a processing module 1102 based on an interactive attention mechanism, and a feedforward neural network 1103.

[0146] According to embodiments of this disclosure, the deep learning model 1100 can be a Transformer Decoder. A processing module 1101 based on a self-attention mechanism processes the first sample point feature sequence to achieve interaction of its own feature information. Then, a processing module 1102 based on an interactive attention mechanism processes the first sample point feature sequence and the second sample point feature sequence to achieve interaction between their feature information. Finally, a feedforward neural network 1103 performs feature mapping to obtain the third sample point sequence.

[0147] According to embodiments of this disclosure, the processing module 1101 based on the self-attention mechanism can be constructed according to the following formula (1):

[0148]

[0149] Where X represents the feature matrix of the feature sequence of the first sample point; X T d represents the transpose of X; k The dimension representing the point feature.

[0150] According to embodiments of this disclosure, the processing module 1102 based on the interactive attention mechanism can be constructed according to the following formula (2):

[0151]

[0152] Where Q represents the feature matrix obtained after the feature sequence of the first sample points is processed by the self-attention mechanism-based processing module 1101; K and V represent the feature matrices obtained by the feature sequence of the second sample points through different mapping functions; d k Dimension representing point features 。

[0153] According to embodiments of this disclosure, the feedforward neural network 1103 can be constructed based on any mapping function that first expands and then shrinks the dimension of the point features, without any specific limitation.

[0154] Figure 12 The illustration shows a schematic diagram of the principle of a training method for a deep learning model according to an embodiment of the present disclosure.

[0155] like Figure 12 As shown, in step 1200, the first sample point feature sequence 1221 and the second sample point feature sequence 1222 are jointly input into the deep learning model 1201. The second sample point feature sequence is the sample label of the first sample point feature sequence. After the deep learning model 1201 processes the first sample point feature sequence 1221 and the second sample point feature sequence 1222, a third sample point feature sequence is obtained. The third sample point feature sequence 1223 is mapped one-to-one with the second sample point feature sequence 1222, and a loss value 1224 is obtained based on the loss function. It is then determined whether the loss value 1224 has converged. If the loss value 1224 has not converged, the model parameters are updated. The deep learning model 1201 is adjusted using the updated model parameters 1225, and the deep learning model 1201 is retrained until the loss value 1224 converges, resulting in a trained deep learning model.

[0156] According to embodiments of this disclosure, during the training process of a deep learning model, each sample point in the third sample point feature sequence corresponds to the first sample point feature sequence and the second sample point feature sequence.

[0157] For example, the first sample point feature sequence 1221 may include features of sample point M1, features of sample point M2, ..., features of sample point M n The features of the second sample point feature sequence 1222 may include the label features of sample point M1, the label features of sample point M2, ..., sample point M n The label features.

[0158] According to an embodiment of this disclosure, during the training of the deep learning model, the first sample point feature sequence 1221 is processed by the deep learning model 1201 to obtain the features of sample point T1 in the third sample point feature sequence. The label features of sample point M1 in the first sample point feature sequence 1221 and the second sample point feature sequence 1222 are processed by the deep learning model 1201 to obtain the features of sample point T2 in the third sample point feature sequence. This process continues until all sample point features of the first sample point feature sequence 1221 and the second sample point feature sequence 1222 are processed using the deep learning model, at which point a stop symbol is output. Therefore, the number of sample points in the third sample point feature sequence 1223 is the same as the number of sample points in the first sample point feature sequence 1221 and the second sample point feature sequence 1222.

[0159] According to embodiments of this disclosure, the loss value obtained based on the loss function can be generated by producing an average loss value based on the loss values ​​corresponding to the S third sample points. Alternatively, the loss value can be obtained by configuring different weights for different third sample points, based on the loss values ​​corresponding to the S third sample points and the weights corresponding to the third sample points.

[0160] According to embodiments of this disclosure, the loss function can be configured according to actual business needs, and is not limited thereto. For example, the loss function may include at least one of the following: cross-entropy loss function, exponential loss function, and squared loss function.

[0161] Figure 13 A block diagram of a map generation apparatus according to an embodiment of the present disclosure is shown schematically.

[0162] like Figure 13 As shown, the map generation device 1300 includes: a first generation module 1301, an extraction module 1302, a first acquisition module 1303, and a second generation module 1304.

[0163] The first generation module 1301 is used to generate a fused feature map based on image data and point cloud data.

[0164] Extraction module 1302 is used to extract the first point feature sequence of the fused feature map.

[0165] The first acquisition module 1303 is used to process the first point feature sequence and the fused feature map based on an attention strategy to obtain target element information.

[0166] The second generation module 1304 is used to generate a map based on the target feature information.

[0167] According to embodiments of this disclosure, the first generation module 1301 includes a first generation submodule, a second generation submodule, and a third generation submodule.

[0168] The first generation submodule is used to generate an image feature map based on a predetermined feature map, a first parameter, and image features of the image data. The predetermined feature map includes random features that only follow a Gaussian distribution, and the first parameter represents the device parameters used to acquire the image data.

[0169] The second generation submodule is used to generate a point cloud feature map based on a predetermined feature map, point cloud features of point cloud data, and a second parameter. The second parameter represents the device parameters used to collect point cloud data.

[0170] The third generation submodule is used to fuse the image feature map and the point cloud feature map to generate a fused feature map.

[0171] According to embodiments of this disclosure, the extraction module 1302 includes: a first obtaining submodule, a second obtaining submodule, and a third obtaining submodule.

[0172] The first acquisition submodule is used to perform semantic segmentation processing on the fused feature map to obtain a semantic feature map.

[0173] The second submodule is used to perform instance segmentation processing on the semantic feature map to obtain the instance feature map.

[0174] The third submodule is used to extract the first point feature sequence from the instance feature map.

[0175] According to embodiments of this disclosure, the third obtaining submodule includes: a first obtaining unit, an extraction unit, and a first generating unit.

[0176] The first obtaining unit is used to process the instance feature map to obtain a discrete point set, which includes multiple discrete points and feature information of multiple discrete points.

[0177] The extraction unit is used to extract a target discrete point sequence from multiple discrete points based on the feature information of multiple discrete points. The target discrete point sequence includes multiple target discrete points and their feature information.

[0178] The first generation unit is used to generate a first point feature sequence based on the feature information of multiple target discrete points.

[0179] According to embodiments of this disclosure, the first obtaining unit includes a first obtaining subunit and a second obtaining subunit.

[0180] The first sub-unit is used to perform erosion processing on the instance feature map to obtain a sparse instance feature map.

[0181] The second sub-unit is used to perform pooling on the sparse instance feature map to obtain a discrete point set.

[0182] According to embodiments of this disclosure, the first extraction unit includes a first determining subunit, a third obtaining subunit, and an extraction subunit.

[0183] The first determining sub-unit is used to determine the discrete point sorting strategy based on the feature information of multiple discrete points.

[0184] The third sub-unit is used to sort multiple discrete points in a forward order according to the discrete point sorting strategy to obtain the first discrete point sequence.

[0185] Extract sub-units to extract the target discrete point sequence from the first discrete point sequence based on a selection strategy.

[0186] According to embodiments of this disclosure, an extraction sub-unit is used to perform sparsification on a first discrete point sequence to obtain a first sparse point sequence. Following a sorting strategy, the first sparse point sequence is reverse-sorted to obtain a second sparse point sequence. The second sparse point sequence is then subjected to sparsification to obtain the target discrete point sequence.

[0187] According to an embodiment of this disclosure, the first discrete point sequence includes I discrete points, where I is an integer greater than 1. Sparse processing is performed on the first discrete point sequence to obtain a first sparse point sequence, including: for the i-th discrete point among the I discrete points, deleting k discrete points to obtain a second discrete point sequence, wherein the pixel distance between each of the k discrete points and the i-th discrete point satisfies a preset threshold range, i is greater than or equal to 1 and less than or equal to 1, and k is an integer greater than or equal to 1. The j-th discrete point is extracted from the second discrete point sequence as a sparse point, and i is incremented, where j is an integer greater than or equal to 1. If i is determined to be less than 1, the operation of deleting k discrete points is returned; if i is determined to be equal to 1, the first sparse point sequence is obtained.

[0188] According to embodiments of this disclosure, the first feature sequence includes feature information of M points, where M is an integer greater than 1, and the target element information includes feature information of M+N points, where N is an integer greater than 1. The first obtaining module includes a fourth obtaining submodule, a fourth generating submodule, and a fifth generating submodule.

[0189] The fourth submodule is used to process the first point feature sequence and the fused feature map based on the attention strategy to obtain the target feature information of the M+n point, where n is an integer greater than or equal to 1 and less than or equal to N.

[0190] The fourth generation submodule is used to generate the second point feature sequence based on the target feature information of the M+n point and the feature sequence of the first point, and increment n.

[0191] The fifth generation submodule is used to perform attention-based processing operations on the second point feature sequence and fused feature map when n is less than N; and to generate target feature information based on the target feature information of M+N points when n is equal to N.

[0192] According to embodiments of this disclosure, the fourth obtaining submodule includes a second obtaining unit, a second generating unit, and a third obtaining unit.

[0193] The second acquisition unit is used to process the first point feature sequence based on the self-attention mechanism to obtain the first feature information of the M+nth point.

[0194] The second generation unit is used to generate the third point feature sequence based on the first point feature sequence and the first feature information of the M+n point.

[0195] The third acquisition unit is used to process the third point feature sequence and the fused feature map based on the interactive attention mechanism to obtain the target feature information of the M+nth feature point.

[0196] Figure 14 A block diagram of a deep learning model training apparatus according to an embodiment of the present disclosure is shown schematically.

[0197] like Figure 14 As shown, the deep learning model training device 1400 includes a second acquisition module 1401 and a third acquisition module 4102.

[0198] The second obtaining module 1401 is used to obtain a first sample point feature sequence and a second sample point feature sequence based on the sample image, wherein the first sample point feature sequence includes S first sample points, the second sample point feature sequence includes S second sample points, S is an integer greater than 1, the sample image represents a fused feature map generated based on image data and point cloud data, and the s-th second sample point in the second sample point feature sequence is the sample label of the s-th first sample point in the first sample point feature sequence, where s is an integer greater than or equal to 1 and less than or equal to S; and

[0199] The third acquisition module 1402 is used to train a deep learning model using the feature sequences of the first and second sample points to obtain a trained deep learning model.

[0200] According to embodiments of this disclosure, the third obtaining module 1402 includes a fifth obtaining submodule, a sixth obtaining submodule, and a seventh obtaining submodule.

[0201] The fifth submodule is used to process the feature sequences of the first and second sample points based on an attention strategy to obtain the feature sequence of the third sample points, which includes S third sample points.

[0202] The sixth submodule is used to generate the sth loss value based on the loss function, according to the sth third sample point of the third sample point feature sequence and the sth second sample point of the second sample point feature sequence.

[0203] The seventh submodule is used to adjust the model parameters of the deep learning model based on the loss values ​​corresponding to the S third sample points, so as to obtain the trained deep learning model.

[0204] According to embodiments of this disclosure, the seventh obtaining submodule includes a third generating unit and an adjusting unit.

[0205] The third generation unit is used to generate the average loss value based on the loss values ​​corresponding to the S third sample points.

[0206] The adjustment unit is used to adjust the model parameters of the deep learning model based on the average loss value.

[0207] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0208] According to an embodiment of the present disclosure, an electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method described above.

[0209] According to embodiments of the present disclosure, a non-transitory computer-readable storage medium stores computer instructions, wherein the computer instructions are used to cause a computer to perform the method described above.

[0210] According to an embodiment of this disclosure, a computer program product includes a computer program that, when executed by a processor, implements the method described above.

[0211] Figure 15A schematic block diagram of an example electronic device 1500 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0212] like Figure 15 As shown, device 1500 includes a computing unit 1501, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 1502 or a computer program loaded into random access memory (RAM) 1503 from storage unit 1508. The RAM 1503 may also store various programs and data required for the operation of device 1500. The computing unit 1501, ROM 1502, and RAM 1503 are interconnected via bus 1504. Input / output (I / O) interface 1505 is also connected to bus 1504.

[0213] Multiple components in device 1500 are connected to I / O interface 1505, including: input unit 1506, such as keyboard, mouse, etc.; output unit 1507, such as various types of monitors, speakers, etc.; storage unit 1508, such as disk, optical disk, etc.; and communication unit 1509, such as network card, modem, wireless transceiver, etc. Communication unit 1509 allows device 1500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0214] The computing unit 1501 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1501 performs the various methods and processes described above, such as map generation methods or deep learning model training methods. For example, in some embodiments, the map generation method or deep learning model training method can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1508. In some embodiments, part or all of the computer program can be loaded and / or installed on device 1500 via ROM 1502 and / or communication unit 1509. When the computer program is loaded into RAM 1503 and executed by the computing unit 1501, one or more steps of the map generation method or deep learning model training method described above can be performed. Alternatively, in other embodiments, computing unit 1501 may be configured by any other suitable means (e.g., by means of firmware) to perform map generation methods or training methods for deep learning models.

[0215] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0216] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0217] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0218] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0219] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0220] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, distributed system servers, or servers incorporating blockchain technology.

[0221] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0222] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A map generation method, comprising: Generate a fused feature map based on image data and point cloud data; The first point feature sequence in the fused feature map is extracted through semantic segmentation; The first feature sequence includes feature information of M points, where M is an integer greater than 1; the target element information includes feature information of M+N points, where N is an integer greater than 1. Based on the attention strategy, the first point feature sequence and the fused feature map are processed to obtain the target feature information of the (M+n)th point, where n is an integer greater than or equal to 1 and less than or equal to N; Based on the target feature information of the (M+n)th point and the feature sequence of the first point, a feature sequence of the second point is generated, and n is incremented. If n is determined to be less than N, then for the second point feature sequence and the fused feature map, the processing operation based on the attention strategy is returned; and Given that n equals N, the target element information is generated based on the target feature information of M+N points; and A map is generated based on the target element information.

2. The method according to claim 1, wherein, The step of generating a fused feature map based on image data and point cloud data includes: An image feature map is generated based on a predetermined feature map, a first parameter, and the image features of the image data. The predetermined feature map includes random features that follow only a Gaussian distribution. The first parameter represents the device parameters used to acquire the image data. A point cloud feature map is generated based on the predetermined feature map, the point cloud features of the point cloud data, and a second parameter, wherein the second parameter represents the device parameters used to acquire the point cloud data; and The image feature map and the point cloud feature map are fused together to generate the fused feature map.

3. The method according to claim 1, wherein, The step of extracting the first point feature sequence from the fused feature map through semantic segmentation includes: The fused feature map is subjected to semantic segmentation processing to obtain a semantic feature map; The semantic feature map is subjected to instance segmentation processing to obtain an instance feature map; and Extract the first point feature sequence from the instance feature map.

4. The method according to claim 3, wherein, Extracting the first point feature sequence from the instance feature map includes: The instance feature map is processed to obtain a discrete point set, which includes multiple discrete points and feature information of the multiple discrete points; Based on the feature information of the plurality of discrete points, a sequence of target discrete points is extracted from the plurality of discrete points, wherein the sequence of target discrete points includes the plurality of target discrete points and the feature information of the plurality of target discrete points; and Based on the feature information of the multiple target discrete points, a first point feature sequence is generated.

5. The method according to claim 4, wherein, The process of processing the instance feature map to obtain a discrete point set includes: The instance feature map is eroded to obtain a sparse instance feature map; and The sparse instance feature map is pooled to obtain the discrete point set.

6. The method according to claim 4, wherein, The step of extracting a target discrete point sequence from multiple discrete points based on the feature information of the discrete points includes: Based on the feature information of the multiple discrete points, a discrete point sorting strategy is determined; According to the discrete point sorting strategy, the multiple discrete points are sorted in a forward order to obtain a first discrete point sequence; and Based on the selection strategy, the target discrete point sequence is extracted from the first discrete point sequence.

7. The method according to claim 6, wherein, The step of extracting the target discrete point sequence from the first discrete point sequence based on the selection strategy includes: The first discrete point sequence is subjected to sparse processing to obtain the first sparse point sequence; According to the sorting strategy, the first sparse point sequence is reversed to obtain the second sparse point sequence; and The second sparse point sequence is subjected to the sparse processing to obtain the target discrete point sequence.

8. The method according to claim 7, wherein, The first discrete point sequence includes I discrete points, where I is an integer greater than 1; the step of performing sparse processing on the first discrete point sequence to obtain the first sparse point sequence includes: For the i-th discrete point among the I discrete points, delete k discrete points to obtain a second discrete point sequence, wherein the pixel distance between each of the k discrete points and the i-th discrete point satisfies a preset threshold range, i is greater than or equal to 1 and less than or equal to 1, and k is an integer greater than or equal to 1. Extract the j-th discrete point from the second discrete point sequence as a sparse point, and increment by i, where j is an integer greater than or equal to 1; and If i is less than 1, return to the operation of deleting k discrete points; if i is equal to 1, obtain the first sparse point sequence.

9. The method according to claim 1, wherein, The process of processing the first point feature sequence and the fused feature map based on the attention strategy to obtain the target feature information of the (M+n)th point includes: Based on the self-attention mechanism, the feature sequence of the first point is processed to obtain the first feature information of the M+nth point; Based on the first point feature sequence and the first feature information of the (M+n)th point, generate the third point feature sequence; and Based on the interactive attention mechanism, the third point feature sequence and the fused feature map are processed to obtain the target feature information of the M+nth feature point.

10. A method for training a deep learning model, comprising: Based on the sample image, a first sample point feature sequence and a second sample point feature sequence are obtained, wherein the first sample point feature sequence includes S first sample points, and the second sample point feature sequence includes S second sample points, where S is an integer greater than 1. The sample image represents a fused feature map generated based on image data and point cloud data. The s-th second sample point in the second sample point feature sequence is the sample label of the s-th first sample point in the first sample point feature sequence, where s is an integer greater than or equal to 1 and less than or equal to S. Based on an attention strategy, the first sample point feature sequence is processed to obtain the features of the first sample point in the third sample feature sequence; the first sample point feature sequence and the second sample point feature sequence are processed to obtain the features of the second sample point in the third sample point feature sequence, until all sample point features in the first sample point feature sequence and the second sample point feature sequence are processed to obtain the third sample point feature sequence; the third sample point feature sequence includes S third sample points. Based on the loss function, the s-th loss value is generated according to the s-th third sample point in the feature sequence of the third sample points and the s-th second sample point in the feature sequence of the second sample points; and The model parameters of the deep learning model are adjusted according to the loss values ​​corresponding to the S third sample points to obtain the trained deep learning model.

11. The method according to claim 10, wherein, The step of adjusting the model parameters of the deep learning model based on the loss values ​​corresponding to the S third sample points includes: Based on the loss values ​​corresponding to the S third sample points, an average loss value is generated; and The model parameters of the deep learning model are adjusted based on the average loss value.

12. A map generation apparatus, comprising: The first generation module is used to generate a fused feature map based on image data and point cloud data; The extraction module is used to extract the first point feature sequence of the fused feature map through semantic segmentation; the first point feature sequence includes feature information of M points, where M is an integer greater than 1, and the target element information includes feature information of M+N points, where N is an integer greater than 1. The first acquisition module includes: The fourth submodule is used to process the first point feature sequence and the fused feature map based on an attention strategy to obtain the target feature information of the (M+n)th point, where n is an integer greater than or equal to 1 and less than or equal to N; The fourth generation submodule is used to generate a second point feature sequence based on the target feature information of the (M+n)th point and the feature sequence of the first point, and increment n. The fifth generation submodule is used to, when n is determined to be less than N, return and perform processing operations based on the attention strategy for the second point feature sequence and the fused feature map; and when n is determined to be equal to N, generate the target element information based on the target feature information of M+N points; and The second generation module is used to generate a map based on the target element information.

13. The apparatus according to claim 12, wherein, The first generation module includes: The first generation submodule is used to generate an image feature map based on a predetermined feature map, a first parameter, and the image features of the image data. The predetermined feature map includes random features that only follow a Gaussian distribution, and the first parameter represents the device parameters used to acquire the image data. The second generation submodule is used to generate a point cloud feature map based on the predetermined feature map, the point cloud features of the point cloud data, and a second parameter, wherein the second parameter represents the device parameters used to acquire the point cloud data; and The third generation submodule is used to fuse the image feature map and the point cloud feature map to generate the fused feature map.

14. The apparatus according to claim 12, wherein, The extraction module includes: The first obtaining submodule is used to perform semantic segmentation processing on the fused feature map to obtain a semantic feature map; The second obtaining submodule is used to perform instance segmentation processing on the semantic feature map to obtain an instance feature map; and The third obtaining submodule is used to extract the first point feature sequence from the instance feature map.

15. The apparatus of claim 14, wherein the third obtaining submodule comprises: The first obtaining unit is used to process the instance feature map to obtain a discrete point set, the discrete point set including multiple discrete points and feature information of the multiple discrete points; An extraction unit is configured to extract a sequence of target discrete points from the plurality of discrete points based on the feature information of the plurality of discrete points, wherein the sequence of target discrete points includes the plurality of target discrete points and the feature information of the plurality of target discrete points; as well as The first generation unit is used to generate a first point feature sequence based on the feature information of the plurality of target discrete points.

16. The apparatus of claim 15, wherein the first obtaining unit comprises: The first obtaining subunit is used to perform erosion processing on the instance feature map to obtain a sparse instance feature map; as well as The second obtaining subunit is used to perform pooling processing on the sparse instance feature map to obtain the discrete point set.

17. The apparatus according to claim 15, wherein the extraction unit comprises: The first determining subunit is used to determine the discrete point sorting strategy based on the feature information of the multiple discrete points; The third obtaining subunit is used to sort the multiple discrete points in a forward order according to the discrete point sorting strategy to obtain the first discrete point sequence. as well as Extraction sub-units are used to extract the target discrete point sequence from the first discrete point sequence based on a selection strategy.

18. The apparatus of claim 17, wherein the extraction subunit is configured to: The first discrete point sequence is subjected to sparse processing to obtain the first sparse point sequence; According to the discrete point sorting strategy, the first sparse point sequence is sorted in reverse order to obtain the second sparse point sequence; and The second sparse point sequence is subjected to the sparse processing to obtain the target discrete point sequence.

19. The apparatus according to claim 18, wherein the first discrete point sequence comprises I discrete points, where I is an integer greater than 1; the step of performing sparse processing on the first discrete point sequence to obtain a first sparse point sequence comprises: For the i-th discrete point among the I discrete points, delete k discrete points to obtain a second discrete point sequence, wherein the pixel distance between each of the k discrete points and the i-th discrete point satisfies a preset threshold range, i is greater than or equal to 1 and less than or equal to 1, and k is an integer greater than or equal to 1. Extract the j-th discrete point from the second discrete point sequence as a sparse point, and increment by i, where j is an integer greater than or equal to 1; and If i is less than 1, return to the operation of deleting k discrete points; if i is equal to 1, obtain the first sparse point sequence.

20. The apparatus according to claim 12, wherein, The fourth obtaining submodule includes: The second acquisition unit is used to process the first point feature sequence based on the self-attention mechanism to obtain the first feature information of the M+n point; The second generation unit is configured to generate a third point feature sequence based on the first point feature sequence and the first feature information of the M+nth point; and The third acquisition unit is used to process the third point feature sequence and the fused feature map based on the interactive attention mechanism to obtain the target feature information of the M+nth feature point.

21. A training device for a deep learning model, comprising: The second obtaining module is used to obtain a first sample point feature sequence and a second sample point feature sequence based on the sample image, wherein the first sample point feature sequence includes S first sample points, the second sample point feature sequence includes S second sample points, S is an integer greater than 1, the sample image represents a fused feature map generated based on image data and point cloud data, and the s-th second sample point in the second sample point feature sequence is the sample label of the s-th first sample point in the first sample point feature sequence, s is an integer greater than or equal to 1 and less than or equal to S; and The third acquisition module includes: The fifth submodule is used to process the first sample point feature sequence based on an attention strategy to obtain the features of the first sample point in the third sample point feature sequence; to process the first sample point feature sequence and the second sample point feature sequence to obtain the features of the second sample point in the third sample point feature sequence, until all sample point features in the first sample point feature sequence and the second sample point feature sequence have been processed to obtain the third sample point feature sequence; the third sample point feature sequence includes S third sample points; The sixth submodule is used to generate the s-th loss value based on the loss function, according to the s-th third sample point of the third sample point feature sequence and the s-th second sample point of the second sample point feature sequence; and The seventh submodule is used to adjust the model parameters of the deep learning model according to the loss values ​​corresponding to the S third sample points, so as to obtain the trained deep learning model.

22. The apparatus according to claim 21, wherein, The seventh obtaining submodule includes: The third generation unit is used to generate an average loss value based on the loss values ​​corresponding to the S third sample points; and An adjustment unit is used to adjust the model parameters of the deep learning model based on the average loss value.

23. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-9 or 10-11.

24. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-9 or 10-11.

25. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-9 or 10-11.

Images