Semantic representation processing method and device, electronic equipment and computer storage medium
By extracting features from image and LiDAR data, performing geometric attention fusion and semantic tagging, and combining masking, the problem of insufficient feature fusion in autonomous driving is solved, and the robustness and performance of the model are improved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING PHIGENT TECHNOLOGY CO LTD
- Filing Date
- 2023-02-02
- Publication Date
- 2026-05-08
AI Technical Summary
In existing autonomous driving technologies, the pipeline paradigm and the end-to-end driving paradigm have shortcomings in feature fusion methods, which leads to performance limitations in urban end-to-end driving, especially the loss of traffic light information at a distance.
By extracting features from image and LiDAR data, performing geometric attention fusion and semantic tagging, and combining masking, semantic fusion and reconstruction of features from top-view and radar bird's-eye view are achieved, thereby improving the robustness of autonomous driving.
It improves the performance and robustness of autonomous driving models under normal and incomplete/corrupted sensor data conditions, while maintaining semantic representations of more detailed information.
Smart Images
Figure CN116152765B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of autonomous driving technology, and in particular to a semantic representation processing method, a semantic representation processing apparatus, an electronic device, and a computer-readable storage medium. Background Technology
[0002] The latest trends in autonomous driving tasks can be divided into two paradigms: pipelined and end-to-end. The pipelined paradigm breaks down driving into sequential modular tasks, mainly including localization, scene reconstruction, path planning, and driving control. The end-to-end driving paradigm often uses state-action imitation learning or reinforcement learning to mediate feature representations of states to teach the agent to act correctly in a given driving environment.
[0003] However, pipeline paradigms are often designed for specific perception tasks rather than a comprehensive, holistic representation of the state. Therefore, current 3D driving feature fusion methods are not entirely suitable for comprehensive end-to-end driving tasks.
[0004] In addition, the end-to-end driving paradigm requires separate branch processing in different modalities, and then the final fusion is completed through attention. However, this fusion is a pure geometric fusion, which will hinder the performance of comprehensive and complex urban end-to-end driving, because geometric transformation and network downsampling may lose the special system information of autonomous driving, such as traffic lights in the distance. Summary of the Invention
[0005] In view of the above problems, embodiments of the present invention are proposed to provide a semantic representation processing method, a semantic representation processing apparatus, an electronic device, and a computer-readable storage medium that overcome or at least partially solve the above problems.
[0006] To address the aforementioned problems, embodiments of the present invention disclose a semantic representation processing method, the method comprising:
[0007] Features are extracted from the image to obtain the planar view features, and features are extracted from the lidar data to obtain the lidar features;
[0008] Geometric attention fusion is performed on the plan view features and the radar features to obtain plan view features and radar bird's-eye view features;
[0009] The plan view features are divided into multiple plan view feature blocks, and the multiple plan view feature blocks are marked as semantic tags; the radar bird's-eye view features are divided into multiple radar bird's-eye view feature blocks, and the multiple radar bird's-eye view feature blocks are marked as semantic tags.
[0010] Based on the semantic tags, the top view feature blocks and the radar bird's-eye view feature blocks are semantically fused to obtain a semantic representation;
[0011] Based on the mask, the semantic representation is reconstructed to obtain the reconstructed semantic representation.
[0012] In one or more embodiments, the geometric attention fusion of the plan view features and the radar features to obtain plan view features and radar bird's-eye view features includes:
[0013] The monotonic-to-bird's-eye view conversion attention is used to perform geometric attention fusion on the plan view features and the radar features to obtain the plan view features and radar bird's-eye view features.
[0014] In one or more embodiments, the step of performing geometric attention fusion on the plan view features and the radar features using monotonic-to-bird's-eye view transformation attention to obtain plan view features and radar bird's-eye view features includes:
[0015] The plan view features are segmented to obtain multiple plan view feature vectors, and the radar features are segmented to obtain multiple radar feature vectors.
[0016] The multiple plan view feature vectors and the multiple radar feature vectors are geometrically fused to obtain the fused feature vector;
[0017] The fused feature vectors are mapped to top-view features and radar bird's-eye view features respectively through an attention mechanism.
[0018] In one or more embodiments, semantic fusion is performed on the top-view feature block and the radar bird's-eye view feature block based on the semantic tagging to obtain a semantic representation, including:
[0019] An encoder with position embedding is used to spatially align the semantic tags of the plan view feature block and the radar bird's-eye view feature block in a preset semantic space to obtain the aligned semantic representation.
[0020] In one or more embodiments, the step of performing semantic reconstruction based on a mask to obtain a reconstructed semantic representation includes:
[0021] The semantic representation is masked using a mask at a preset ratio to obtain a semantic representation that includes the masked portion;
[0022] The covered portion is reconstructed to obtain the reconstructed semantic representation.
[0023] Accordingly, embodiments of the present invention also disclose a semantic representation processing apparatus, the apparatus comprising:
[0024] The extraction module is used to extract features from images to obtain planar view features, and to extract features from LiDAR data to obtain radar features;
[0025] A conversion module is used to perform geometric attention fusion on the plan view features and the radar features to obtain plan view features and radar bird's-eye view features;
[0026] The tagging module is used to cut the plan view features into multiple plan view feature blocks and tag the multiple plan view feature blocks as semantic tags, and to cut the radar bird's-eye view features into multiple radar bird's-eye view feature blocks and tag the multiple radar bird's-eye view feature blocks as semantic tags;
[0027] An alignment module is used to semantically fuse the top view feature block and the radar bird's-eye view feature block based on the semantic tags to obtain a semantic representation;
[0028] The reconstruction module is used to perform semantic reconstruction on the semantic representation based on the mask, so as to obtain the reconstructed semantic representation.
[0029] In one or more embodiments, the conversion module is specifically used for:
[0030] The monotonic-to-bird's-eye view conversion attention is used to perform geometric attention fusion on the plan view features and the radar features to obtain the plan view features and radar bird's-eye view features.
[0031] In one or more embodiments, the conversion module is specifically used for:
[0032] The plan view features are segmented to obtain multiple plan view feature vectors, and the radar features are segmented to obtain multiple radar feature vectors.
[0033] The multiple plan view feature vectors and the multiple radar feature vectors are geometrically fused to obtain the fused feature vector;
[0034] The fused feature vectors are mapped to top-view features and radar bird's-eye view features respectively through an attention mechanism.
[0035] In one or more embodiments, the alignment module is specifically used for:
[0036] An encoder with position embedding is used to spatially align the semantic tags of the plan view feature block and the radar bird's-eye view feature block in a preset semantic space to obtain the aligned semantic representation.
[0037] In one or more embodiments, the reconstruction module is specifically used for:
[0038] The semantic representation is masked using a mask at a preset ratio to obtain a semantic representation that includes the masked portion;
[0039] The covered portion is reconstructed to obtain the reconstructed semantic representation.
[0040] Accordingly, embodiments of the present invention disclose an electronic device, including: a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program, when executed by the processor, implements the various steps of the above-described semantic representation processing method embodiments.
[0041] Accordingly, embodiments of the present invention disclose a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the various steps of the above-described semantic representation processing method embodiments.
[0042] The embodiments of the present invention have the following advantages:
[0043] Features are extracted from images to obtain planar view features, and features are extracted from LiDAR data to obtain radar features. Geometric attention fusion is performed on the planar view features and radar features to obtain planar view features and radar bird's-eye view features. The planar view features are segmented into multiple planar view feature blocks, and these blocks are labeled with semantic tags. Similarly, the radar bird's-eye view features are segmented into multiple radar bird's-eye view feature blocks, and these blocks are labeled with semantic tags. Semantic fusion is performed on the planar view feature blocks and radar bird's-eye view feature blocks based on the semantic tags to obtain a semantic representation. Semantic reconstruction is performed on the semantic representation based on a mask to obtain the reconstructed semantic representation. Furthermore, after embedding a unified semantic representation of different modalities, a masking process similar to that used in language models is introduced. That is, a portion of the semantic representation is masked, and then the semantic representation is enhanced by having the model complete the masked portion. This forces the unified semantic representation to retain more detailed information. At the same time, this masking process can not only improve the model's performance under normal driving conditions, but also enhance the model's robustness in autonomous driving under conditions of incomplete sensor data or damaged sensors. Attached Figure Description
[0044] Figure 1 This is a flowchart illustrating the steps of an embodiment of a semantic representation processing method according to the present invention;
[0045] Figure 2 This is a schematic diagram of the MBT structure according to an embodiment of the present invention;
[0046] Figure 3 This is a schematic diagram of the semantic representation after masking according to an embodiment of the present invention;
[0047] Figure 4 This is the feature fusion network structure of an embodiment of the present invention;
[0048] Figure 5 This is a structural block diagram of an embodiment of a semantic representation processing device according to the present invention. Detailed Implementation
[0049] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0050] One of the core concepts of this invention is to extract features from images to obtain planar view features, and to extract features from LiDAR data to obtain radar features; to perform geometric attention fusion on the planar view features and the radar features to obtain planar view features and radar bird's-eye view features; to divide the planar view features into blocks to obtain multiple planar view feature blocks, and to label the multiple planar view feature blocks as semantic tags; and to divide the radar bird's-eye view features into blocks to obtain multiple radar bird's-eye view feature blocks, and to label the multiple radar bird's-eye view feature blocks as semantic tags; to perform semantic fusion on the planar view feature blocks and the radar bird's-eye view feature blocks based on the semantic tags to obtain a semantic representation; and to perform semantic reconstruction on the semantic representation based on a mask to obtain a reconstructed semantic representation. This invention proposes a novel end-to-end multimodal autonomous driving perception network framework. This framework can combine the advantages of geometric fusion and semantic fusion, embedding individual and joint representations of each modality into a unified semantic space. Furthermore, by embedding a unified semantic representation across different modalities, a masking process similar to that used in language models is introduced. This involves masking a portion of the semantic representation and then enhancing it by having the model complete the masked part. This forces the unified semantic representation to retain more detailed information. Simultaneously, this masking process not only improves the model's performance under normal driving conditions but also enhances its robustness in autonomous driving scenarios with incomplete or damaged sensor data.
[0051] Reference Figure 1 The diagram illustrates a flowchart of an embodiment of a semantic representation processing method according to the present invention, which may specifically include the following steps:
[0052] Step 101: Extract features from the image to obtain the planar view features, and extract features from the lidar data to obtain the lidar features.
[0053] The image can be a planar image captured by a camera in the vehicle. After the image is captured, planar view features can be extracted from the image using a neural network, such as a CNN (Convolutional Neural Network). Of course, other neural networks or models besides CNN can also be used to extract planar view features. In practical applications, the settings can be configured according to actual needs, and this embodiment of the invention does not impose any restrictions on this.
[0054] Furthermore, during image acquisition, images can be captured using a single camera or multiple cameras, and then stitched together to form a monochromatic view. For example, three images can be captured using three cameras, each with a 60° FOV (Field of View), and then stitched together to form a monochromatic view, which is then reshaped into a 3x160x704 shape. Of course, other acquisition methods are also applicable to this embodiment of the invention. In practical applications, settings can be configured according to actual needs, and this embodiment of the invention does not impose any limitations on this.
[0055] Meanwhile, the LiDAR in the vehicle can also collect LiDAR data, and then extract radar features from the LiDAR data. For example, PointPillar can be used for feature extraction, or other methods can be used. In practical applications, the settings can be configured according to actual needs, and this embodiment of the invention does not impose any limitations on this.
[0056] Step 102: Perform geometric attention fusion on the plan view features and the radar features to obtain plan view features and radar bird's-eye view features.
[0057] After obtaining the top-view features and radar features, geometric attention fusion can be performed on them to obtain top-view features and radar BEV (Bird Eye's View) features. Moreover, since the lower-level features still maintain strong geometric relationships, a tight BEV feature representation can be extracted with less interference, thus fully utilizing the advantages of geometric fusion.
[0058] In this embodiment of the invention, the geometric attention fusion of the plan view features and the radar features to obtain plan view features and radar bird's-eye view features includes:
[0059] The monotonic-to-bird's-eye view conversion attention is used to perform geometric attention fusion on the plan view features and the radar features to obtain the plan view features and radar bird's-eye view features.
[0060] Specifically, when fusing top view features and radar features, a monotonic to BEV translation attention mechanism (MBT) can be used. MBT can help enrich each modality through cross-modal processing.
[0061] The step of using monotonic-to-bird's-eye view transformation attention to perform geometric attention fusion on the plan view features and the radar features to obtain plan view features and radar bird's-eye view features includes:
[0062] The plan view features are segmented to obtain multiple plan view feature vectors, and the radar features are segmented to obtain multiple radar feature vectors.
[0063] The multiple plan view feature vectors and the multiple radar feature vectors are geometrically fused to obtain the fused feature vector;
[0064] The fused feature vectors are mapped to top-view features and radar bird's-eye view features respectively through an attention mechanism.
[0065] Figure 2 The diagram illustrates the structure of MBT. Specifically, the planar view features are vertically segmented to obtain multiple planar view feature vectors. Simultaneously, the radar features are segmented to obtain multiple radar feature vectors. These multiple planar view feature vectors and radar feature vectors are then geometrically fused to obtain a fused feature vector. A multi-head attention mechanism is then used to map the fused feature vector back to planar view features and radar BEV features, respectively. This mapping can be from a monocular 120 FOV to BEV features. Since the monocular field of view is limited, the BEV feature can be a trapezoidal shape. Therefore, a single camera parameter matrix can be used to sample features within the 120 FOV.
[0066] Step 103: Slice the plan view features into multiple plan view feature blocks and mark the multiple plan view feature blocks as semantic tags; and slice the radar bird's-eye view features into multiple radar bird's-eye view feature blocks and mark the multiple radar bird's-eye view feature blocks as semantic tags.
[0067] After obtaining the planar view features and radar BEV features, the planar view features can be segmented into multiple planar view feature blocks. Each planar view feature block is then labeled with a semantic tag, resulting in multiple planar view feature blocks with semantic tags. Similarly, the radar BEV features can be segmented into multiple radar BEV feature blocks, and each radar BEV feature block is then labeled with a semantic tag, resulting in multiple radar BEV feature blocks with semantic tags.
[0068] Step 104: Based on the semantic tags, perform semantic fusion on the top view feature block and the radar bird's-eye view feature block to obtain a semantic representation.
[0069] After obtaining multiple plan view feature blocks and multiple radar BEV feature blocks, the two can be spatially aligned in a preset, unified semantic space based on semantic tokens to obtain semantic representations, thereby realizing multimodal semantic representation fusion on the token representation in a unified semantic space.
[0070] In this embodiment of the invention, the step of semantically fusing the plan view feature block and the radar bird's-eye view feature block based on the semantic tag to obtain a semantic representation includes:
[0071] An encoder with position embedding is used to spatially align the semantic tags of the plan view feature block and the radar bird's-eye view feature block in a preset semantic space to obtain the aligned semantic representation.
[0072] Specifically, an encoder with position embedding can be used to spatially align the semantic tags of the plan view feature block and the radar BEV feature block in a preset semantic space. At the same time, masked tokens can be spatially aligned with the two and encoded using an encoder to obtain the aligned semantic representation, which is the semantic representation after semantic fusion.
[0073] Step 105: Perform semantic reconstruction on the semantic representation based on the mask to obtain the reconstructed semantic representation.
[0074] To enhance the joint fusion of various modalities in the semantic representation, after obtaining the semantic representation, partial semantic representations can be semantically reconstructed using a mask to obtain the reconstructed semantic representation.
[0075] In this embodiment of the invention, the step of performing semantic reconstruction based on a mask to obtain the reconstructed semantic representation includes:
[0076] The semantic representation is masked using a mask at a preset ratio to obtain a semantic representation that includes the masked portion;
[0077] The covered portion is reconstructed to obtain the reconstructed semantic representation.
[0078] Specifically, a portion of the semantic representation can be masked using a preset ratio, such as 25% or 50%, to obtain a semantic representation containing the masked portion. Then, the masked portion is reconstructed to obtain the reconstructed semantic representation. For example, ... Figure 3As shown, the semantic representation is enhanced by covering up 50% of the image and then having the model complete the covered portion. This forces the unified semantic representation to retain more detailed information.
[0079] After obtaining the semantic representation of the semantic reconstruction, the decoder can predict the vehicle's trajectory based on the Seg Head, Decoder Head, and BEV Head.
[0080] For ease of understanding, Figure 4 The feature fusion network structure of an embodiment of the present invention is illustrated. Two independent ResNet32 residual networks can be used to extract planar view features and LiDAR features. Then, geometric attention fusion is performed on the planar view features and LiDAR features based on MBT to obtain planar view features and LiDAR BEV features. The planar view features and LiDAR BEV features are then segmented and labeled to obtain multiple planar view feature blocks and multiple LiDAR BEV feature blocks. Semantic fusion alignment and encoding are performed on each planar view feature block and LiDAR BEV feature block in a unified semantic space to obtain a semantic representation. Finally, masking-reconstruction is performed on the semantic representation based on a mask to obtain the final semantic representation.
[0081] In this embodiment of the invention, features are extracted from images to obtain planar view features, and features are extracted from LiDAR data to obtain radar features; geometric attention fusion is performed on the planar view features and the radar features to obtain planar view features and radar bird's-eye view features; the planar view features are segmented to obtain multiple planar view feature blocks, and the multiple planar view feature blocks are marked as semantic tags; the radar bird's-eye view features are segmented to obtain multiple radar bird's-eye view feature blocks, and the multiple radar bird's-eye view feature blocks are marked as semantic tags; the planar view feature blocks and the radar bird's-eye view feature blocks are semantically fused based on the semantic tags to obtain a semantic representation; semantic reconstruction is performed on the semantic representation based on a mask to obtain a reconstructed semantic representation. This embodiment of the invention proposes a novel end-to-end multimodal autonomous driving perception network framework. This framework can combine the advantages of geometric fusion and semantic fusion, embedding individual and joint representations of each modality into a unified semantic space. Furthermore, by embedding a unified semantic representation across different modalities, a masking process similar to that used in language models is introduced. This involves masking a portion of the semantic representation and then enhancing it by having the model complete the masked part. This forces the unified semantic representation to retain more detailed information. Simultaneously, this masking process not only improves the model's performance under normal driving conditions but also enhances its robustness in autonomous driving scenarios with incomplete or damaged sensor data.
[0082] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.
[0083] Reference Figure 5 The diagram illustrates a structural block diagram of an embodiment of a semantic representation processing apparatus according to the present invention, which may specifically include the following modules:
[0084] The extraction module 501 is used to extract features from the image to obtain the planar view features, and to extract features from the lidar data to obtain the lidar features;
[0085] The conversion module 502 is used to perform geometric attention fusion on the plan view features and the radar features to obtain plan view features and radar bird's-eye view features;
[0086] The tagging module 503 is used to cut the plan view features into multiple plan view feature blocks and tag the multiple plan view feature blocks as semantic tags, and to cut the radar bird's-eye view features into multiple radar bird's-eye view feature blocks and tag the multiple radar bird's-eye view feature blocks as semantic tags.
[0087] Alignment module 504 is used to semantically fuse the top view feature block and the radar bird's-eye view feature block based on the semantic tag to obtain a semantic representation;
[0088] The reconstruction module 505 is used to perform semantic reconstruction on the semantic representation based on the mask to obtain the reconstructed semantic representation.
[0089] In this embodiment of the invention, the conversion module is specifically used for:
[0090] The monotonic-to-bird's-eye view conversion attention is used to perform geometric attention fusion on the plan view features and the radar features to obtain the plan view features and radar bird's-eye view features.
[0091] In this embodiment of the invention, the conversion module is specifically used for:
[0092] The plan view features are segmented to obtain multiple plan view feature vectors, and the radar features are segmented to obtain multiple radar feature vectors.
[0093] The multiple plan view feature vectors and the multiple radar feature vectors are geometrically fused to obtain the fused feature vector;
[0094] The fused feature vectors are mapped to top-view features and radar bird's-eye view features respectively through an attention mechanism.
[0095] In this embodiment of the invention, the alignment module is specifically used for:
[0096] An encoder with position embedding is used to spatially align the semantic tags of the plan view feature block and the radar bird's-eye view feature block in a preset semantic space to obtain the aligned semantic representation.
[0097] In this embodiment of the invention, the reconstruction module is specifically used for:
[0098] The semantic representation is masked using a mask at a preset ratio to obtain a semantic representation that includes the masked portion;
[0099] The covered portion is reconstructed to obtain the reconstructed semantic representation.
[0100] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.
[0101] This invention also provides an electronic device, comprising:
[0102] It includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, it implements the various processes of the above-described semantic representation processing method embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here.
[0103] This invention also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the above-described semantic representation processing method embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here.
[0104] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.
[0105] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0106] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0107] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0108] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0109] Although preferred embodiments of the present invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present invention.
[0110] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.
[0111] The present invention has provided a detailed description of a semantic representation processing method and a semantic representation processing apparatus. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.
Claims
1. A method for processing semantic representation, characterized in that, The method includes: Features are extracted from the image to obtain the planar view features, and features are extracted from the lidar data to obtain the lidar features; Geometric attention fusion is performed on the plan view features and the radar features to obtain fused plan view features and radar bird's-eye view features; The fused plan view features are divided into multiple plan view feature blocks, and the multiple plan view feature blocks are marked as semantic tags. Similarly, the radar bird's-eye view features are divided into multiple radar bird's-eye view feature blocks, and the multiple radar bird's-eye view feature blocks are marked as semantic tags. Based on the semantic tags, the top view feature blocks and the radar bird's-eye view feature blocks are semantically fused to obtain a semantic representation; Based on the mask, semantic reconstruction is performed on the semantic representation to obtain the reconstructed semantic representation; The geometric attention fusion of the plan view features and the radar features to obtain the fused plan view features and radar bird's-eye view features includes: The monotonic-to-bird's-eye view conversion attention is used to perform geometric attention fusion on the plan view features and the radar features to obtain the fused plan view features and radar bird's-eye view features.
2. The semantic representation processing method according to claim 1, characterized in that, The step involves using monotonic-to-bird's-eye view transformation attention to perform geometric attention fusion on the plan view features and the radar features, resulting in fused plan view features and radar bird's-eye view features, including: The plan view features are segmented to obtain multiple plan view feature vectors, and the radar features are segmented to obtain multiple radar feature vectors. The multiple plan view feature vectors and the multiple radar feature vectors are geometrically fused to obtain the fused feature vector; The fused feature vectors are mapped to fused top-view features and radar bird's-eye view features respectively through an attention mechanism.
3. The semantic representation processing method according to claim 1, characterized in that, The semantic fusion of the top-view feature block and the radar bird's-eye view feature block based on the semantic tagging to obtain a semantic representation includes: An encoder with position embedding is used to spatially align the semantic tags of the plan view feature block and the radar bird's-eye view feature block in a preset semantic space to obtain the aligned semantic representation.
4. The semantic representation processing method according to claim 1, characterized in that, The semantic reconstruction based on the mask to obtain the reconstructed semantic representation includes: The semantic representation is masked using a mask at a preset ratio to obtain a semantic representation that includes the masked portion; The covered portion is reconstructed to obtain the reconstructed semantic representation.
5. A semantic representation processing apparatus, characterized in that, The device includes: The extraction module is used to extract features from images to obtain planar view features, and to extract features from LiDAR data to obtain radar features; A conversion module is used to perform geometric attention fusion on the plan view features and the radar features to obtain fused plan view features and radar bird's-eye view features; The tagging module is used to cut the fused plan view features into multiple plan view feature blocks and tag the multiple plan view feature blocks as semantic tags; and to cut the radar bird's-eye view features into multiple radar bird's-eye view feature blocks and tag the multiple radar bird's-eye view feature blocks as semantic tags. An alignment module is used to semantically fuse the top view feature block and the radar bird's-eye view feature block based on the semantic tags to obtain a semantic representation; The reconstruction module is used to perform semantic reconstruction on the semantic representation based on the mask, so as to obtain the reconstructed semantic representation; The conversion module is specifically used for: The monotonic-to-bird's-eye view conversion attention is used to perform geometric attention fusion on the plan view features and the radar features to obtain the fused plan view features and radar bird's-eye view features.
6. The semantic representation processing apparatus according to claim 5, characterized in that, The conversion module is specifically used for: The plan view features are segmented to obtain multiple plan view feature vectors, and the radar features are segmented to obtain multiple radar feature vectors. The multiple plan view feature vectors and the multiple radar feature vectors are geometrically fused to obtain the fused feature vector; The fused feature vectors are mapped to fused top-view features and radar bird's-eye view features respectively through an attention mechanism.
7. The semantic representation processing apparatus according to claim 5, characterized in that, The alignment module is specifically used for: An encoder with position embedding is used to spatially align the semantic tags of the plan view feature block and the radar bird's-eye view feature block in a preset semantic space to obtain the aligned semantic representation.
8. The semantic representation processing apparatus according to claim 5, characterized in that, The reconstruction module is specifically used for: The semantic representation is masked using a mask at a preset ratio to obtain a semantic representation that includes the masked portion; The covered portion is reconstructed to obtain the reconstructed semantic representation.
9. An electronic device, characterized in that, include: A processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program, when executed by the processor, implements the steps of the processing method for semantic representation as described in any one of claims 1 to 4.
10. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, and when executed by a processor, the computer program implements the steps of the processing method for semantic representation as described in any one of claims 1 to 4.