Semantic environment processing methods, devices and electronic equipment for autonomous driving
By extracting features from vehicle sensor data and fusing historical environment data, combined with attention computing, an accurate semantic environment can be constructed in real time, solving the problem of low safety in autonomous driving caused by offline construction and improving the safety and adaptability of autonomous driving.
Patent Information
- Application Number
- CN202410502364.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-04-24
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2044-04-24
AI Technical Summary
In existing technologies, the semantic environment built offline cannot be updated in real time, resulting in low safety for autonomous driving and an inability to adapt to different weather conditions and complex scenarios.
By extracting environmental features from environmental data collected by vehicle-mounted sensors, combining historical global semantic environment and short-term memory feature fusion, and utilizing attention computing and semantic extraction processing, an accurate semantic environment is constructed in real time.
It enables the construction of accurate and effective semantic environments under different weather conditions and complex scenarios, thereby improving the safety and adaptability of autonomous driving.
Smart Images

Figure CN118609070B_ABST
Abstract
Description
Technical Field
[0001] This application relates to vehicle technology, and more particularly to a method, apparatus, electronic device, computer-readable storage medium, and computer program product for processing the semantic environment of autonomous driving. Background Technology
[0002] The rapid development of autonomous driving technology has had a profound impact on the transportation sector, particularly on traffic safety, traffic flow management, and urban planning. As a key component of autonomous driving technology, the vehicle-centric semantic environment provides essential environmental perception information for autonomous vehicles. This semantic environment includes various environmental elements such as road geometry, traffic signals, pedestrian walkways, and parking spaces, all of which are crucial for safe driving.
[0003] Currently, the industry widely adopts offline semantic environment construction methods. However, this method relies on manual annotation, which has disadvantages such as high cost and delayed updates. Since it cannot achieve real-time updates, when used for autonomous driving, the offline (historical) semantic environment is prone to inconsistency with the actual semantic environment, resulting in low safety of autonomous driving. Summary of the Invention
[0004] This application provides a semantic environment processing method, apparatus, electronic device, computer-readable storage medium, and computer program product for autonomous driving, which can construct an accurate and effective semantic environment online, thereby helping to improve the safety of autonomous driving.
[0005] The technical solution of this application is implemented as follows:
[0006] This application provides a semantic environment processing method for autonomous driving, including:
[0007] Environmental features are extracted from the environmental data collected by the vehicle-mounted sensors at time t to obtain the real-time environmental features at time t.
[0008] Extract the historical local semantic environment corresponding to the vehicle's position at time t from the historical global semantic environment, and convert the historical local semantic environment corresponding to the vehicle's position at time t into historical environment features at time t; wherein, the historical global semantic environment includes historical local semantic environments corresponding to multiple positions respectively;
[0009] The real-time environmental features at time t, the historical environmental features at time t, and the fused environmental features at time t-1 are subjected to feature fusion processing to obtain the fused environmental features at time t.
[0010] Attention is calculated on the fused environment features at time t based on the optimized environment features at time t-1 to obtain the optimized environment features at time t.
[0011] Semantic extraction processing is performed on the optimized environment features at time t to obtain the real-time local semantic environment at time t.
[0012] This application provides a semantic environment processing device for autonomous driving, comprising:
[0013] The perception module is used to extract environmental features from the environmental data collected by the vehicle-mounted sensors at time t to obtain the real-time environmental features at time t.
[0014] The extraction module is used to extract the historical local semantic environment corresponding to the vehicle position at time t from the historical global semantic environment, and convert the historical local semantic environment corresponding to the vehicle position at time t into historical environment features at time t; wherein, the historical global semantic environment includes historical local semantic environments corresponding to multiple positions respectively;
[0015] The feature fusion module is used to perform feature fusion processing on the real-time environmental features at time t, the historical environmental features at time t, and the fused environmental features at time t-1 to obtain the fused environmental features at time t.
[0016] The attention calculation module is used to perform attention calculation on the fused environment features at time t based on the optimized environment features at time t-1, so as to obtain the optimized environment features at time t.
[0017] The semantic extraction module is used to perform semantic extraction processing on the optimized environment features at time t to obtain the real-time local semantic environment at time t.
[0018] This application provides an electronic device, including:
[0019] Memory, used to store executable instructions;
[0020] The processor, when executing executable instructions stored in the memory, implements the semantic environment processing method for autonomous driving provided in this application.
[0021] This application provides a computer-readable storage medium storing executable instructions for inducing a processor to execute, thereby implementing the semantic environment processing method for autonomous driving provided in this application.
[0022] This application provides a computer program product including executable instructions for inducing a processor to execute and implement the semantic environment processing method for autonomous driving provided in this application.
[0023] This application has the following beneficial effects:
[0024] This application performs environmental feature extraction processing on the environmental data collected by the vehicle-mounted sensor at time t to obtain the real-time environmental features at time t; extracts the historical local semantic environment corresponding to the vehicle's position at time t from the historical global semantic environment, and converts the historical local semantic environment corresponding to the vehicle's position at time t into historical environmental features at time t; performs feature fusion processing on the real-time environmental features at time t, the historical environmental features at time t, and the fused environmental features at time t-1 to obtain the fused environmental features at time t; performs attention calculation on the fused environmental features at time t based on the optimized environmental features at time t-1 to obtain the optimized environmental features at time t; and performs semantic extraction processing on the optimized environmental features at time t to obtain the real-time local semantic environment at time t. This application mimics human memory mechanisms, using the historical global semantic environment as long-term memory and the fused and optimized environmental features as short-term memory. This combined approach of long-term and short-term memory improves the effectiveness of online semantic environment construction, enhancing adaptability to different weather conditions and complex scenarios. Simultaneously, attention-based computation captures key information from features while ignoring invalid and redundant information, resulting in a more accurate and effective semantic environment. In summary, this application can construct accurate and effective semantic environments online, contributing to improved safety in autonomous driving. Attached Figure Description
[0025] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0026] Figure 1 This is a schematic diagram of the architecture of the semantic environment processing system for autonomous driving provided in an embodiment of this application;
[0027] Figure 2 This is a schematic diagram of the structure of the vehicle-mounted device provided in an embodiment of this application;
[0028] Figure 3A This is a first flowchart illustrating the semantic environment processing method for autonomous driving provided in this application embodiment;
[0029] Figure 3B This is a second flowchart illustrating the semantic environment processing method for autonomous driving provided in this application embodiment;
[0030] Figure 3CThis is a schematic diagram of the third process of the semantic environment processing method for autonomous driving provided in the embodiments of this application;
[0031] Figure 4 This is a schematic diagram of a bird's-eye view coordinate system provided in an embodiment of this application;
[0032] Figure 5 This is a schematic diagram of the fourth process of the semantic environment processing method for autonomous driving provided in the embodiments of this application;
[0033] Figure 6 This is a schematic diagram of the semantic environment processing model provided in the embodiments of this application. Detailed Implementation
[0034] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0035] In the following description, references to "some embodiments" describe a subset of all possible embodiments; however, it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict. In the following description, the term "a plurality of" means at least two.
[0036] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.
[0037] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.
[0038] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.
[0039] 1) Semantic Environment: Also known as the autonomous driving semantic environment, it describes the environment surrounding the vehicle. The semantic environment includes various environmental elements, such as stop lines, lane lines, and curbs. In the embodiments of this application, the semantic environment can be represented by vectorization. For example, various environmental elements can be classified into two vector categories: lines (such as lane lines) and surfaces (such as zebra crossings). In this way, the stored semantic environment is independent of the internal structure of the model (referring to the semantic environment processing model), which has high interpretability and maintainability; at the same time, it takes into account the advantages of low storage cost and rich information.
[0040] The semantic environment involved in the embodiments of this application includes a local semantic environment and a global semantic environment. The local semantic environment is observed from a specific location, while the global semantic environment is obtained by fusing the local semantic environments corresponding to multiple locations. For example, the local semantic environments corresponding to multiple locations can be mapped from the bird's-eye view coordinate system to the global coordinate system, and then spliced in the global coordinate system to obtain the global semantic environment.
[0041] It is worth noting that since the global semantic environment is usually pre-built, it can also be referred to as the historical global semantic environment in this embodiment of the application.
[0042] 2) Coordinate system: used to describe the position and trajectory of an object in space. Through the coordinate system, abstract geometric concepts can be transformed into specific numerical calculations, thus facilitating the solution of various mathematical and physical problems.
[0043] The coordinate systems involved in this application include a bird's-eye view (BEV) coordinate system and a global coordinate system. The bird's-eye view is a perspective of viewing an object or scene from above, like a bird looking down at the ground from the air. When constructing the bird's-eye view coordinate system, the projection of the vehicle's front axle onto the road plane can be used as the origin. The x-axis describes the vehicle's left and right position information, and the y-axis describes the vehicle's front and rear position information. The global coordinate system refers to the Earth coordinate system. Compared to the bird's-eye view coordinate system, each coordinate in the global coordinate system is unique in the real world. In this application embodiment, since the bird's-eye view coordinate system is centered on the vehicle, the mapping between the bird's-eye view coordinate system and the global coordinate system can be achieved based on the vehicle's pose.
[0044] 3) Vehicle-mounted sensors: These refer to sensors mounted on a vehicle for sensing the vehicle's surrounding environment, such as vehicle-mounted cameras and vehicle-mounted radar. In the embodiments of this application, a vehicle-mounted sensor may refer to a single sensor or a combination of multiple sensors.
[0045] This application provides a semantic environment processing method, apparatus, electronic device, computer-readable storage medium, and computer program product for autonomous driving, which can construct an accurate and effective semantic environment online, helping to improve the safety of autonomous driving. The following describes exemplary applications of the electronic device provided in this application. The electronic device provided in this application can be implemented as an in-vehicle device or as a server.
[0046] See Figure 1 , Figure 1 This is an architectural diagram of an autonomous driving semantic environment processing system 100 provided in an embodiment of this application. The vehicle-mounted device 400 is connected to the server 200 through the network 300. The vehicle-mounted device 400 is a general term for electronic devices installed in the vehicle, which are used to provide functions such as data calculation, braking control, communication control, and driving assistance.
[0047] In some embodiments, taking an in-vehicle device as an example, the semantic environment processing method for autonomous driving provided in this application can be implemented by an in-vehicle device. For example, the in-vehicle device 400 performs environmental feature extraction processing on the environmental data collected by the in-vehicle sensors at time t to obtain the real-time environmental features at time t; it extracts the historical local semantic environment corresponding to the vehicle's position at time t from the historical global semantic environment, and converts the historical local semantic environment corresponding to the vehicle's position at time t into historical environmental features at time t; wherein, the historical global semantic environment includes historical local semantic environments corresponding to multiple positions respectively; it performs feature fusion processing on the real-time environmental features at time t, the historical environmental features at time t, and the fused environmental features at time t-1 to obtain the fused environmental features at time t; it performs attention calculation on the fused environmental features at time t based on the optimized environmental features at time t-1 to obtain the optimized environmental features at time t; and it performs semantic extraction processing on the optimized environmental features at time t to obtain the real-time local semantic environment at time t. Here, the t-th time can be the current time. When the on-board device 400 constructs the real-time local semantic environment at the t-th time, it performs planning processing based on the real-time local semantic environment at the t-th time to obtain the driving path, performs decision processing based on the real-time local semantic environment at the t-th time to obtain the driving decision, and controls the vehicle according to the driving path and the driving decision to achieve autonomous driving. The above method can be implemented without network connectivity and has a wide range of applications.
[0048] In some embodiments, taking the electronic device as a server as an example, the semantic environment processing method for autonomous driving provided in this application embodiment can also be implemented by a server. For example, the vehicle-mounted device 400 can send the environmental data at time t to the server 200. The server 200 performs environmental feature extraction processing on the environmental data at time t to obtain the real-time environmental features at time t; it extracts the historical local semantic environment corresponding to the vehicle position at time t (which can also be sent to the server 200 by the vehicle-mounted device 400) from the historical global semantic environment, and converts the historical local semantic environment corresponding to the vehicle position at time t into historical environmental features at time t; it performs feature fusion processing on the real-time environmental features at time t, the historical environmental features at time t, and the fused environmental features at time t-1 to obtain the fused environmental features at time t; it performs attention calculation on the fused environmental features at time t based on the optimized environmental features at time t-1 to obtain the optimized environmental features at time t; and it performs semantic extraction processing on the optimized environmental features at time t to obtain the real-time local semantic environment at time t. Here, the t-th time point can be the current time. When the server 200 constructs the real-time local semantic environment at the t-th time point, it performs planning processing based on this environment to obtain a driving path, performs decision processing to obtain a driving decision, and determines control commands based on the driving path and driving decision. These control commands are then sent to the vehicle-mounted device 400, enabling the device to control the vehicle. Alternatively, the server 200 can also send the real-time local semantic environment at the t-th time point to the vehicle-mounted device 400, allowing it to perform subsequent operations. This method delegates some operations to the server 200, reducing the computational and storage burden on the vehicle-mounted device 400.
[0049] In some embodiments, the in-vehicle device 400 or server 200 can implement the semantic environment processing method for autonomous driving provided in this application embodiment by running a computer program. For example, the computer program can be a native program or software module in an operating system; it can be a native application (APP), i.e., a program that needs to be installed in the operating system to run; it can also be a mini-program, i.e., a program that only needs to be downloaded to a browser environment to run; or it can be a mini-program that can be embedded in any APP, and the mini-program can be controlled by the user to run or close. In short, the above-mentioned computer program can be any form of application, module, or plugin.
[0050] In some embodiments, server 200 may be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.
[0051] Taking the electronic device provided in this application as an example, which is an in-vehicle device, it can be understood that in the case where the electronic device is a server, Figure 2 Some parts of the structure shown (such as the user interface, presentation module, and input processing module) can be omitted. See also Figure 2 , Figure 2 This is a schematic diagram of the structure of the vehicle-mounted device 400 provided in the embodiments of this application. Figure 2 The illustrated vehicle-mounted device 400 includes at least one processor 410, a memory 450, at least one network interface 420, and a user interface 430. The various components in the vehicle-mounted device 400 are coupled together via a bus system 440. It is understood that the bus system 440 is used to implement communication between these components. In addition to a data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2 The general labeled all buses as Bus System 440.
[0052] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.
[0053] User interface 430 includes one or more output devices 431 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.
[0054] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices physically located away from the processor 410.
[0055] The memory 450 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 450 described in this application embodiment is intended to include any suitable type of memory.
[0056] In some embodiments, memory 450 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.
[0057] Operating system 451 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;
[0058] The network communication module 452 is used to reach other computing devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.
[0059] Presentation module 453 is configured to enable the presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 431 (e.g., a display screen, a speaker, etc.) associated with user interface 430;
[0060] The input processing module 454 is used to detect and translate one or more user inputs or interactions from one or more input devices 432.
[0061] In some embodiments, the semantic environment processing device for autonomous driving provided in this application can be implemented in software. Figure 2 A semantic environment processing device 455 for autonomous driving, stored in memory 450, is shown. This device can be software in the form of programs and plugins, and includes the following software modules: a perception module 4551, an extraction module 4552, a feature fusion module 4553, an attention calculation module 4554, and a semantic extraction module 4555. These modules are logically connected and can therefore be arbitrarily combined or further divided according to their implemented functions. The functions of each module will be described below.
[0062] The semantic environment processing method for autonomous driving provided in this application will be described in conjunction with exemplary applications and implementations of the electronic devices provided in the embodiments of this application.
[0063] See Figure 3A , Figure 3A This is a flowchart illustrating a semantic environment processing method for autonomous driving provided in an embodiment of this application, which will be combined with... Figure 3A The steps shown are explained.
[0064] In step 101, environmental feature extraction processing is performed on the environmental data collected by the vehicle-mounted sensor at time t to obtain the real-time environmental features at time t.
[0065] Here, environmental data collected by the vehicle-mounted sensors at time t is acquired. The vehicle-mounted sensors are used to collect data (i.e., perceive) about the vehicle's surrounding environment. Time t can represent any time, such as the current time. Then, environmental feature extraction processing is performed on the environmental data at time t, that is, key and effective information is extracted from the environmental data at time t to obtain the real-time environmental features at time t. In this embodiment, environmental feature extraction processing can be implemented based on the principles of Artificial Neural Networks (ANNs). For example, environmental feature extraction processing can be performed on the environmental data at time t through specific network layers to obtain the real-time environmental features at time t. It is worth noting that the model and network (referring to artificial neural networks) involved in this embodiment have the same meaning.
[0066] In some embodiments, the vehicle-mounted sensor is a vehicle-mounted camera, and the environmental data is image data. The above-mentioned environmental feature extraction processing of the environmental data collected by the vehicle-mounted sensor at time t can be achieved in the following way: image feature extraction processing is performed on the image data at time t to obtain the image features at time t; the image features at time t are mapped from the image space to the bird's-eye view space to obtain the real-time environmental features at time t.
[0067] Here, the vehicle-mounted sensor is the vehicle-mounted camera, and the environmental data is image data. Based on this, image feature extraction processing is performed on the image data at time t to obtain the image features at time t. For example, image feature extraction processing can be performed on the image data at time t using networks such as ResNet or VGG. Considering that the semantic environment is centered on the vehicle, it needs to be described using a bird's-eye view coordinate system (bird's-eye view space). Since the image features at time t are located in image space, the image features at time t are mapped from image space to bird's-eye view space to obtain the real-time environmental features at time t. Because the bird's-eye view provides global environmental perception and can better reflect the position of objects in space, mapping the image features at time t from image space to bird's-eye view space can enhance spatial perception, making the real-time environmental features at time t more accurately reflect the surrounding environment of the vehicle.
[0068] In some embodiments, the vehicle-mounted camera can be a surround-view camera, thereby enabling the collected environmental data to cover all angles around the vehicle, improving its comprehensiveness and detail.
[0069] In some embodiments, the above-mentioned mapping of image features at time t from image space to bird's-eye view space to obtain real-time environmental features at time t can be achieved by performing any of the following processes: mapping image features at time t from image space to bird's-eye view space using camera parameters of the vehicle-mounted camera to obtain real-time environmental features at time t; mapping image features at time t from image space to bird's-eye view space using a spatial mapping model to obtain real-time environmental features at time t; wherein the spatial mapping model learns the mapping relationship between image space and bird's-eye view space.
[0070] Here, two methods are provided for mapping from image space to bird's-eye view space:
[0071] 1) Inverse Projective Mapping (IPM) is performed using the camera parameters of the onboard camera to map the image features at time t from the image space to the bird's-eye view space, thus obtaining the real-time environmental features at time t. The camera parameters can include intrinsic and extrinsic parameters. Since camera parameters are usually fixed or calibrated at the vehicle's factory, the mapping process in method 1) can be performed quickly, meeting the real-time requirements of autonomous driving; furthermore, the camera parameters themselves are relatively accurate, ensuring the accuracy of the mapping process.
[0072] 2) The image features at time t are mapped from the image space to the bird's-eye view space using a spatial mapping model to obtain the real-time environmental features at time t. The spatial mapping model learns the mapping relationship between the image space and the bird's-eye view space. Here, a model learning approach is used to learn the mapping relationship between the image space and the bird's-eye view space, thereby mapping the image features at time t to the real-time environmental features at time t. The type of spatial mapping model is not limited; for example, it could be a View Parsing Network (VPN). The spatial mapping model in method 2) is established by learning a large number of mapping relationships between the image space and the bird's-eye view space, thus adapting to various complex road scenes and changing environmental conditions; at the same time, it also reduces dependence on camera parameters, making it more flexible.
[0073] In step 102, the historical local semantic environment corresponding to the vehicle position at time t is extracted from the historical global semantic environment, and the historical local semantic environment corresponding to the vehicle position at time t is converted into the historical environment feature at time t; wherein, the historical global semantic environment includes the historical local semantic environments corresponding to multiple positions respectively.
[0074] Here, the pre-stored historical global semantic environment is obtained. The historical global semantic environment is obtained by fusing the historical local semantic environments corresponding to multiple locations. The historical local semantic environments can be constructed in the historical driving process (referring to the historical driving process of the vehicle itself or other vehicles). Given that the vehicle's position at time t is known, the historical local semantic environment corresponding to the vehicle's position at time t is extracted from the historical global semantic environment, and the historical local semantic environment corresponding to the vehicle's position at time t is converted into the historical environment features at time t.
[0075] It is worth noting that, considering that the historical local semantic environment is in vector form rather than feature form, the historical environment features at time t can be extracted from the historical local semantic environment corresponding to the vehicle position at time t through specific network layers, such as several multilayer perceptron (MLP) network layers.
[0076] It is worth noting that the historical global semantic environment is long-term memory (i.e., memory that needs to be stored for a long time). This means that the historical global semantic environment needs to be used at any given time to construct the real-time local semantic environment online. This persistent storage of the historical global semantic environment can be achieved through methods such as database storage.
[0077] In some embodiments, the above-mentioned conversion of the historical local semantic environment corresponding to the vehicle position at time t into historical environmental features at time t can be achieved in the following manner: the historical local semantic environment corresponding to the vehicle position at time t is mapped from the global coordinate system to the bird's-eye view coordinate system; environmental feature extraction processing is performed on the historical local semantic environment corresponding to the vehicle position at time t in the bird's-eye view coordinate system to obtain the historical environmental features at time t.
[0078] Here, the historical local semantic environment is usually described in a bird's-eye view coordinate system. To ensure the accuracy and consistency of the fused historical global semantic environment, it is necessary to map the historical local semantic environments corresponding to multiple locations from the bird's-eye view coordinate system to the global coordinate system, and then concatenate them in the global coordinate system to form the historical global semantic environment. Specifically, the historical local semantic environment can be mapped from the bird's-eye view coordinate system to the global coordinate system based on the pose of the vehicle (the existing historical local semantic environment is based on this vehicle, which may be the same as or different from the aforementioned self-vehicle).
[0079] Correspondingly, the extracted historical local semantic environment corresponding to the vehicle's position at time t is also described in the global coordinate system. Therefore, the historical local semantic environment corresponding to the vehicle's position at time t is mapped from the global coordinate system to the bird's-eye view coordinate system. This mapping can be done based on the vehicle's pose at time t. Then, environmental feature extraction processing is performed on the historical local semantic environment corresponding to the vehicle's position at time t in the bird's-eye view coordinate system. For example, several MLP layers can be used to extract environmental features from the historical local semantic environment corresponding to the vehicle's position at time t in the bird's-eye view coordinate system, resulting in the historical environmental features at time t. This method ensures that the historical environmental features at time t are described in the bird's-eye view coordinate system, thus guaranteeing the consistency of real-time environmental features, historical environmental features, and fused environmental features in the coordinate system.
[0080] In step 103, the real-time environmental features at time t, the historical environmental features at time t, and the fused environmental features at time t-1 are subjected to feature fusion processing to obtain the fused environmental features at time t.
[0081] Here, the fused environmental features at time t are obtained by combining the real-time environmental features at time t, the historical environmental features at time t, and the fused environmental features at time t-1. The method of feature fusion is not limited; for example, it can be a splicing process, or other processing can be performed after the splicing process.
[0082] It is worth noting that the fused environment features are short-term memories. For example, the fused environment features at time t-1 are used to construct the real-time local semantic environment at time t. Once the real-time local semantic environment at time t has been constructed, the fused environment features at time t-1 can be deleted to save storage space. This non-persistent storage of the fused environment features can be achieved through methods such as in-memory storage.
[0083] In step 104, attention is calculated on the fused environment features at time t based on the optimized environment features at time t-1 to obtain the optimized environment features at time t.
[0084] Here, attention is calculated on the fusion environment features at time t based on the optimized environment features at time t-1. That is, the optimized environment features at time t-1 are used as query features (or query vectors) to focus on the important information in the fusion environment features at time t, so as to obtain the optimized environment features at time t. In other words, the optimized environment features at time t can be regarded as the result of further optimization of the fusion environment features at time t.
[0085] It is worth noting that the attention mechanism is a computational model that simulates human attention allocation, improving the understanding and processing of input data by selectively focusing on important information. Attention mechanisms are further subdivided into various types, and this application does not specifically limit the attention mechanism used in its embodiments.
[0086] It's worth noting that the optimized environment features are short-term memory. For example, the optimized environment features at time t-1 are used to construct the real-time local semantic environment at time t. Once the real-time local semantic environment at time t has been constructed, the optimized environment features at time t-1 can be deleted to save storage space. This non-persistent storage of optimized environment features can be achieved through methods such as in-memory storage.
[0087] In step 105, semantic extraction processing is performed on the optimized environment features at time t to obtain the real-time local semantic environment at time t.
[0088] Here, semantic extraction processing is performed on the optimized environment features at time t. For example, semantic extraction processing is performed on the optimized environment features at time t through several MLP layers to regress the point sequence and category of the environment elements. These point sequences with categories constitute the real-time local semantic environment at time t.
[0089] In some embodiments, after semantic extraction processing is performed on the optimized environment features at time t to obtain the real-time local semantic environment at time t, the semantic environment processing method for autonomous driving further includes: updating the historical local semantic environment corresponding to the vehicle position at time t based on the real-time local semantic environment at time t.
[0090] Compared to the historical local semantic environment corresponding to the vehicle's position at time t, the real-time local semantic environment at time t is obtained by referencing more information, meaning it is more accurate. Therefore, the historical local semantic environment corresponding to the vehicle's position at time t can be updated based on the real-time local semantic environment at time t. This means replacing the historical local semantic environment corresponding to the vehicle's position at time t in the historical global semantic environment with the real-time local semantic environment at time t, thereby further improving the accuracy of the historical global semantic environment.
[0091] In some embodiments, when the historical local semantic environment corresponding to the vehicle's position at time t is described by a global coordinate system, the real-time local semantic environment at time t is mapped from the bird's-eye view coordinate system to the global coordinate system. For example, the real-time local semantic environment at time t is mapped from the bird's-eye view coordinate system to the global coordinate system based on the vehicle's pose at time t. Then, the historical local semantic environment corresponding to the vehicle's position at time t in the global coordinate system is updated based on the real-time local semantic environment at time t in the global coordinate system, thereby ensuring the consistency of the update process in the coordinate system.
[0092] In some embodiments, steps 101 to 105 described above can be implemented using a semantic environment processing model. This facilitates deployment, and the effectiveness of semantic environment processing can be further improved through model training. For example, the server can uniformly train the semantic environment processing model and deploy the trained model to the in-vehicle devices of multiple vehicles, enabling these in-vehicle devices to quickly acquire semantic environment processing capabilities.
[0093] like Figure 3A As shown, this embodiment mimics the human memory mechanism, using the historical global semantic environment as long-term memory and the fused and optimized environmental features as short-term memory. This combined long-term and short-term memory improves the effectiveness of online semantic environment construction, enhancing adaptability to different weather conditions and complex scenarios. Simultaneously, attention-based computation captures key information from features while ignoring invalid and redundant information, resulting in a more accurate and effective semantic environment. In summary, this embodiment can construct accurate and effective semantic environments online, contributing to improved safety in autonomous driving.
[0094] In some embodiments, see Figure 3B , Figure 3B This is a flowchart illustrating a semantic environment processing method for autonomous driving provided in an embodiment of this application. Figure 3A Step 104 shown can be implemented through steps 201 to 205, which will be explained in conjunction with each step.
[0095] In step 201, the initial environmental features and the optimized environmental features at time t-1 are fused to obtain the query features.
[0096] Here is an example of the attention mechanism used. First, the pre-initialized initial environment features and the optimized environment features at time t-1 are fused to obtain the query features. The initial environment features learn the relevant rules of attention computation during the initialization process; therefore, fusing the initial environment features improves the efficiency of attention computation.
[0097] In step 202, the query features are processed to extract position offset features, thus obtaining position offset features.
[0098] For example, by performing location offset feature extraction on the query features through several MLP layers, the location offset features are obtained.
[0099] In step 203, the query features are subjected to sampling weight feature extraction processing to obtain sampling weight features.
[0100] For example, sampling weight features can be obtained by performing sampling weight feature extraction on query features through several MLP layers.
[0101] In step 204, the attention feature of the fused environment feature on the position offset feature at time t is calculated based on the query feature.
[0102] Here, deformable attention (DA) processing is performed based on query features, position offset features, and the fused environment features at time t. Specifically, the attention feature of the fused environment features at time t on the position offset features is calculated (collected) based on the query features. Compared to conventional attention mechanisms that assign the same attention to all input positions, the deformable attention mechanism allows for adaptive adjustment of the attention to different positions according to task requirements. This enables better capture of key information in the input data, improving the accuracy of task processing, while ignoring irrelevant or noisy information. This helps to more effectively utilize limited computing resources, improving processing speed and efficiency.
[0103] In step 205, the attention features are weighted according to the sampling weight features to obtain the optimized environment features at time t.
[0104] Here, the sampling weight feature is multiplied by the attention feature, that is, the attention feature is weighted according to the sampling weight feature to obtain the optimization environment feature at time t.
[0105] In some embodiments, the semantic environment processing method for autonomous driving is implemented through a semantic environment processing model. Before performing environmental feature extraction processing on the environmental data collected by the vehicle-mounted sensors at time t, the semantic environment processing method for autonomous driving further includes: acquiring sample environmental data at multiple times and sample local semantic environment at each time; performing feedforward processing on the sample environmental data at each time through the semantic environment processing model to obtain the predicted local semantic environment at each time; calculating the loss value based on the predicted local semantic environment and sample local semantic environment at each time to obtain the loss value; and training the semantic environment processing model and initial environmental features based on the loss value.
[0106] Here, in the case where the semantic environment processing method for autonomous driving is implemented through the semantic environment processing model, the semantic environment processing model and initial environment features can be trained according to the training dataset. In essence, training the semantic environment processing model is updating the parameters of the network layer in the semantic environment processing model, and training the initial environment features is essentially initializing the initial environment features.
[0107] For example, firstly, sample environment data at multiple time points and the sample local semantic environment at each time point are acquired. The vehicle equipped with the onboard sensor used to collect the sample environment data can be the same as or different from the aforementioned self-driving vehicle. The sample local semantic environment can be entirely constructed manually, or it can be manually corrected based on automatic construction to ensure accuracy. Then, the sample environment data at each time point is processed by a semantic environment processing model using feedforward processing to obtain the predicted local semantic environment at each time point (i.e., the predicted real-time local semantic environment). The feedforward processing refers to steps 101 to 105 above and will not be elaborated here. Finally, the predicted local semantic environment at each time point is considered the prediction result, and the sample local semantic environment at each time point is considered the expected result. Loss values are calculated based on the predicted local semantic environment and the sample local semantic environment at each time point, and the semantic environment processing model and initial environment features are trained based on the calculated loss values. The loss function used to calculate the loss value is not limited; for example, it can be the cross-entropy loss function. By using the above methods, we can improve the semantic environment processing effect of the trained semantic environment processing model on the one hand, and learn the relevant rules of attention calculation on the other hand, so that the initial environment features obtained during initialization are more reasonable.
[0108] like Figure 3BAs shown, this embodiment of the application performs feature fusion processing on the initial environmental features and the optimized environmental features at time t-1, thereby incorporating historical experience and making the obtained query features more reasonable. Based on the query features, on the one hand, position offset features are extracted, which can reflect the differences in importance of different positions, helping to focus more on key areas in subsequent processing and improve the efficiency and accuracy of information processing; on the other hand, sampling weight features are extracted, which provide a basis for subsequent weighted processing, that is, fully considering the contribution of different parts of the features. Finally, by calculating attention features and weighted processing, key features can be more accurately focused on, and irrelevant or noisy information can be suppressed, thereby generating more accurate and effective optimized environmental features, which helps to make more accurate and reliable decisions in subsequent tasks.
[0109] In some embodiments, see Figure 3C , Figure 3C This is a flowchart illustrating a semantic environment processing method for autonomous driving provided in an embodiment of this application. Figure 3A After step 105 shown, in step 301, planning processing can be performed based on the real-time local semantic environment at time t to obtain the driving path.
[0110] Here, the real-time local semantic environment at time t can be used in the autonomous driving system to support downstream modules such as planning, decision-making, and control. In the planning module, the driving path is obtained by performing planning processing based on the real-time local semantic environment at time t. For example, under the premise of complying with traffic rules, a safe and efficient driving path is planned based on environmental elements such as road network and lane information in the real-time local semantic environment at time t.
[0111] In step 302, decision processing is performed based on the real-time local semantic environment at time t to obtain the driving decision.
[0112] In the decision-making module, decision processing is performed based on the real-time local semantic environment at time t to obtain a driving decision. For example, appropriate driving decisions are made based on environmental elements such as pedestrians, other vehicles, and traffic lights in the real-time local semantic environment at time t, such as accelerating, decelerating, changing lanes, or stopping.
[0113] In step 303, the vehicle is controlled based on the driving path and driving decisions.
[0114] In the control module, the control commands (such as steering angle, throttle opening, etc.) that should be taken for the vehicle are calculated based on the planned driving path and the driving decisions made. The vehicle is then controlled according to the control commands so that it can drive safely on the driving path according to the driving decisions.
[0115] like Figure 3C As shown, this embodiment of the application performs planning processing based on the real-time local semantic environment at time t to obtain a driving path, performs decision processing based on the real-time local semantic environment at time t to obtain a driving decision, and controls the vehicle based on the driving path and the driving decision. Since the real-time local semantic environment at time t is constructed online, it has the characteristics of real-time and accuracy, thus improving the safety and traffic efficiency of autonomous driving.
[0116] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario. The embodiments of this application are mainly divided into two parts, which will be described separately below.
[0117] 1) Mathematical representation and description of the semantic environment of autonomous driving.
[0118] This application's embodiments involve a bird's-eye view coordinate system and a global coordinate system, such as... Figure 4 As shown, various environmental elements in the autonomous driving semantic environment (such as zebra crossings, lane lines, road arrows, etc.) are represented by a bird's-eye view coordinate system composed of the x-axis, y-axis, and origin O. The origin O can be the projection of the vehicle's front axle onto the road plane. The x-axis describes the vehicle's left and right position information, and the y-axis describes the vehicle's front and rear position information. The global coordinate system refers to the Earth coordinate system. Compared to the bird's-eye view coordinate system, each coordinate in the global coordinate system is unique in the real world. In this embodiment, the mapping between BEV coordinates and global latitude and longitude coordinates can be achieved through vehicle pose and a specific mapping (or projection) method.
[0119] In this embodiment, the semantic environment for autonomous driving is represented using a vectorized approach. For the local semantic environment output by the semantic environment processing model, each environmental element is assigned a category label cls and a series of points. The expression represents, where x i ,y i This represents the BEV coordinates of a point within a certain environmental element, N. p It represents the total number of points that constitute a certain environmental element.
[0120] 2) Construct semantic environments online based on long short-term memory.
[0121] First, let's explain short-term memory and long-term memory separately:
[0122] ① Short-term memory: Short-term memory M at time t s Including the fusion environment features F at time t-1 bev ∈R C ×H×W and the optimization environment characteristics at time t-1. Right now In this system, the short-term memory at time t can be stored in a memory buffer (short-term memory pool) to allow the model to infer the real-time local semantic environment at time t. The purpose of short-term memory is to fuse information from the previous and current time moments, thereby helping the model capture key contextual information and improving its understanding and inference of the local driving scenario at the current moment. This fusion mechanism not only improves the quality of the results at the current moment but also makes the overall system more adaptable and better able to handle changes and uncertainties in the driving scenario. Therefore, short-term memory plays a crucial role in this embodiment, providing an effective reference for modeling driving scenarios across consecutive time moments.
[0123] ②Long-term memory: Long-term memory M l It consists of the final results of historical driving processes. The purpose of long-term memory is to utilize the high-quality environment construction results from historical driving processes to help improve the online model's detection capabilities under adverse conditions such as weather, traffic, and lighting, thereby enhancing the stability of the online semantic environment construction. Long-term memory is stored in an online database (long-term memory pool) and can continuously grow with normal driving. Assume that the existing long-term memory is M'. l The element category label generated by the model when the current vehicle passes through location A at the current time d1. and element point information Then long-term memory will be updated to M l ={M' l ,CLS d1,A ,P d1,A}
[0124] This application provides embodiments such as Figure 5 The diagram shows a flowchart of a semantic environment processing method for autonomous driving, and as shown below. Figure 6 The diagram shown is a structural illustration of a semantic environment processing model, which will combine... Figure 5 as well as Figure 6 This paper describes the process of constructing a real-time local semantic environment online through steps.
[0125] ① Acquire and preprocess the images from the vehicle-mounted surround-view camera acquired at time t. There are no restrictions on the type or number of vehicle-mounted surround-view cameras; similarly, there are no restrictions on the preprocessing method, which may include input size correction, data augmentation, etc.
[0126] ② Use a backbone network to extract image features from the vehicle-mounted surround-view camera image at time t. The type of backbone network is not limited; examples include ResNet and VGG.
[0127] ③ The extracted image features are mapped from the image space to the bird's-eye view space (i.e., BEV transformation) to obtain the real-time environmental features F at time t. bev-cam The mapping method can be either the IPM method, which utilizes camera intrinsic and extrinsic parameters, or the VPN method, which relies on model learning.
[0128] ④ Based on the vehicle's position at time t, a data request is sent to the long-term memory pool. The long-term memory within a certain range around the current vehicle body is extracted and downloaded (corresponding to the historical local semantic environment corresponding to the vehicle's position at time t above). The vectorized long-term memory is processed into a mask form and generated as the historical environment feature F at time t through several MLP layers. bev-l Then, retrieve the fusion environment features at time t-1 from the short-time memory pool. F bev-cam F bev-l as well as Feature fusion processing is performed to obtain the fused environment features at time t. The fusion process can be represented as follows:
[0129]
[0130] Here, concat represents feature fusion at the channel dimension, and GRU represents Gated Recurrent Unit (GRU).
[0131] ⑤ The fused environmental features at time t are stored in the short-term memory pool and propagated to time t+1 for use; on the other hand, the initial environmental features Q are... init The optimized environment feature Q at time t-1 t-1 Perform feature fusion processing and fuse it with the fused environmental features at time t. Attention calculation is performed to obtain the optimized environment features at time t. The calculation process can be represented as follows:
[0132] Q f =concat(Q init Q t-1 )
[0133] O = MLP(Q) f )
[0134] W = MLP(Q) f )
[0135]
[0136] Here, concat refers to fusing vectors along the length dimension to generate the query feature Q. f Q fThe positional offset feature O can be obtained through several MLP layers, and the sampling weight feature W can be obtained through several MLP layers (different from the MLP layers used to calculate the positional offset feature); Q f O After passing through several deformable attention layers (DA), the optimized environment feature Q at time t is obtained by weighting it with W. t .
[0137] ⑥ The optimal environment characteristics Q at time t t On the one hand, it is stored in the short-term memory pool and propagated to the (t+1)th time for use. On the other hand, it outputs the real-time local semantic environment at the tth time through several MLP layers.
[0138] ⑦ The real-time local semantic environment at time t is stored in the long-term memory pool for future use when the vehicle passes through this location (i.e., for constructing a new real-time local semantic environment).
[0139] Inspired by the memory cycle of the human brain, this application divides vehicle historical memory into long-term memory and short-term memory. These two types of memory work together during the current driving process to improve the accuracy and stability of online semantic environment construction. Long-term memory provides macroscopic road condition guidance based on the historical journey, supporting the performance of the semantic environment processing model under adverse driving conditions. Short-term memory performs microscopic fine-tuning by fusing multi-perspective data from multiple moments to quickly capture updates to the environmental elements surrounding the current vehicle position. Online construction of the semantic environment using both long and short-term memory enhances adaptability to adverse weather, abnormal lighting, and complex environments, improving the robustness and effectiveness of autonomous driving systems. This is beneficial for promoting the development of autonomous driving technology, improving traffic safety, and facilitating the construction of intelligent transportation systems.
[0140] The following continues to describe the exemplary structure of the semantic environment processing device 455 for autonomous driving provided in the embodiments of this application as a software module. In some embodiments, such as Figure 2As shown, the software modules stored in the semantic environment processing device 455 for autonomous driving in the memory 450 may include: a perception module 4551, used to perform environmental feature extraction processing on the environmental data collected by the vehicle sensors at time t to obtain the real-time environmental features at time t; and an extraction module 4552, used to extract the historical local semantic environment corresponding to the vehicle's position at time t from the historical global semantic environment, and convert the historical local semantic environment corresponding to the vehicle's position at time t into historical environmental features at time t; wherein, the historical global semantic environment includes multiple locations corresponding to... Historical local semantic environment; Feature fusion module 4553, used to perform feature fusion processing on the real-time environment features at time t, the historical environment features at time t, and the fused environment features at time t-1 to obtain the fused environment features at time t; Attention calculation module 4554, used to perform attention calculation on the fused environment features at time t based on the optimized environment features at time t-1 to obtain the optimized environment features at time t; Semantic extraction module 4555, used to perform semantic extraction processing on the optimized environment features at time t to obtain the real-time local semantic environment at time t.
[0141] In some embodiments, the attention calculation module 4554 is further configured to: perform feature fusion processing on the initial environmental features and the optimized environmental features at time t-1 to obtain query features; perform position offset feature extraction processing on the query features to obtain position offset features; perform sampling weight feature extraction processing on the query features to obtain sampling weight features; calculate the attention features of the fused environmental features at time t on the position offset features based on the query features; and perform weighted processing on the attention features based on the sampling weight features to obtain the optimized environmental features at time t.
[0142] In some embodiments, the semantic environment processing device 455 for autonomous driving is implemented through a semantic environment processing model; the semantic environment processing device 455 for autonomous driving further includes a training module for: acquiring sample environment data at multiple time points and sample local semantic environment at each time point; performing feedforward processing on the sample environment data at each time point through the semantic environment processing model to obtain the predicted local semantic environment at each time point; calculating the loss value based on the predicted local semantic environment at each time point and the sample local semantic environment to obtain the loss value; and training the semantic environment processing model and initial environment features based on the loss value.
[0143] In some embodiments, the vehicle sensor is a vehicle camera, and the environmental data is image data; the perception module 4551 is further configured to: perform image feature extraction processing on the image data at time t to obtain the image features at time t; and map the image features at time t from the image space to the bird's-eye view space to obtain the real-time environmental features at time t.
[0144] In some embodiments, the perception module 4551 is further configured to: perform any of the following processes: map the image features at time t from the image space to the bird's-eye view space using the camera parameters of the vehicle-mounted camera to obtain the real-time environmental features at time t; map the image features at time t from the image space to the bird's-eye view space using a spatial mapping model to obtain the real-time environmental features at time t; wherein the spatial mapping model learns the mapping relationship between the image space and the bird's-eye view space.
[0145] In some embodiments, the historical global semantic environment is described by a global coordinate system; the extraction module 4552 is further configured to: map the historical local semantic environment corresponding to the vehicle position at time t from the global coordinate system to the bird's-eye view coordinate system; and perform environmental feature extraction processing on the historical local semantic environment corresponding to the vehicle position at time t in the bird's-eye view coordinate system to obtain the historical environmental features at time t.
[0146] In some embodiments, the semantic environment processing device 455 for autonomous driving further includes an update module for: updating the historical local semantic environment corresponding to the vehicle position at time t based on the real-time local semantic environment at time t.
[0147] In some embodiments, the semantic environment processing device 455 for autonomous driving further includes a downstream module for: performing planning processing based on the real-time local semantic environment at time t to obtain a driving path; performing decision processing based on the real-time local semantic environment at time t to obtain a driving decision; and controlling the vehicle based on the driving path and the driving decision.
[0148] This application provides a computer program product or computer program that includes executable instructions stored in a computer-readable storage medium. An electronic device's processor reads the executable instructions from the computer-readable storage medium and executes the executable instructions, causing the electronic device to perform the semantic environment processing method for autonomous driving described in this application.
[0149] This application provides a computer-readable storage medium storing executable instructions. When the executable instructions are executed by a processor, the processor will execute the semantic environment processing method for autonomous driving provided in this application.
[0150] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.
[0151] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.
[0152] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborating files (e.g., a file that stores one or more modules, subroutines, or code sections).
[0153] As an example, executable instructions can be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.
[0154] The above are merely embodiments of this application and are not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.
Claims
1. A semantic environment processing method for autonomous driving, characterized in that, include: Environmental features are extracted from the environmental data collected by the vehicle-mounted sensors at time t to obtain the real-time environmental features at time t. Extract the historical local semantic environment corresponding to the vehicle's position at time t from the historical global semantic environment, and convert the historical local semantic environment corresponding to the vehicle's position at time t into historical environment features at time t; wherein, the historical global semantic environment includes historical local semantic environments corresponding to multiple positions respectively; The gated loop unit performs feature fusion processing on the real-time environmental features at time t, the historical environmental features at time t, and the fused environmental features at time t-1 to obtain the fused environmental features at time t. Attention is calculated on the fused environment features at time t based on the optimized environment features at time t-1 to obtain the optimized environment features at time t. Semantic extraction processing is performed on the optimized environment features at time t to obtain the real-time local semantic environment at time t; the real-time local semantic environment includes the point sequence and category of environmental elements; Replace the historical local semantic environment corresponding to the vehicle position at time t in the historical global semantic environment with the real-time local semantic environment at time t.
2. The method according to claim 1, characterized in that, The step of performing attention calculation on the fused environment features at time t based on the optimized environment features at time t-1 to obtain the optimized environment features at time t includes: The initial environmental features and the optimized environmental features at time t-1 are fused to obtain the query features; The query features are subjected to position offset feature extraction processing to obtain position offset features; The query features are subjected to sampling weight feature extraction processing to obtain sampling weight features; Calculate the attention feature of the fused environment feature at time t on the position offset feature based on the query feature; The attention features are weighted according to the sampling weight features to obtain the optimized environment features at time t.
3. The method according to claim 2, characterized in that, The method is implemented through a semantic environment processing model; before performing environmental feature extraction processing on the environmental data collected by the vehicle-mounted sensor at time t, the method further includes: Acquire sample environment data at multiple time points, as well as the local semantic environment of the samples at each time point; The sample environment data at each time step is processed by a semantic environment processing model to obtain the predicted local semantic environment at each time step. The loss value is calculated based on the predicted local semantic environment and the sample local semantic environment at each time step. The semantic environment processing model and initial environment features are trained based on the loss value.
4. The method according to claim 1, characterized in that, The vehicle-mounted sensor is an onboard camera, and the environmental data is image data; the step of extracting environmental features from the environmental data collected by the vehicle-mounted sensor at time t to obtain the real-time environmental features at time t includes: Image feature extraction is performed on the image data at time t to obtain the image features at time t; The image features at time t are mapped from the image space to the bird's-eye view space to obtain the real-time environmental features at time t.
5. The method according to claim 4, characterized in that, The process of mapping the image features at time t from the image space to the bird's-eye view space to obtain the real-time environmental features at time t includes: Perform any of the following processes: The image features at time t are mapped from the image space to the bird's-eye view space using the camera parameters of the vehicle-mounted camera, thus obtaining the real-time environmental features at time t. The image features at time t are mapped from the image space to the bird's-eye view space through a spatial mapping model to obtain the real-time environmental features at time t; the spatial mapping model learns the mapping relationship between the image space and the bird's-eye view space.
6. The method according to claim 1, characterized in that, The historical global semantic environment is described using a global coordinate system; the process of converting the historical local semantic environment corresponding to the vehicle's position at time t into historical environment features at time t includes: Map the historical local semantic environment corresponding to the vehicle's position at time t from the global coordinate system to the bird's-eye view coordinate system; Environmental features are extracted from the historical local semantic environment corresponding to the vehicle's position at time t in the bird's-eye view coordinate system to obtain the historical environmental features at time t.
7. The method according to any one of claims 1 to 6, characterized in that, After performing semantic extraction processing on the optimized environment features at time t to obtain the real-time local semantic environment at time t, the method further includes: The driving path is obtained by planning and processing based on the real-time local semantic environment at time t. The driving decision is obtained by performing decision processing based on the real-time local semantic environment at time t. Control the vehicle based on the driving path and the driving decision.
8. A semantic environment processing device for autonomous driving, characterized in that, include: The perception module is used to extract environmental features from the environmental data collected by the vehicle-mounted sensors at time t to obtain the real-time environmental features at time t. The extraction module is used to extract the historical local semantic environment corresponding to the vehicle position at time t from the historical global semantic environment, and convert the historical local semantic environment corresponding to the vehicle position at time t into historical environment features at time t; wherein, the historical global semantic environment includes historical local semantic environments corresponding to multiple positions respectively; The feature fusion module is used to perform feature fusion processing on the real-time environmental features at time t, the historical environmental features at time t, and the fused environmental features at time t-1 through a gated loop unit to obtain the fused environmental features at time t. The attention calculation module is used to perform attention calculation on the fused environment features at time t based on the optimized environment features at time t-1, so as to obtain the optimized environment features at time t. The semantic extraction module is used to perform semantic extraction processing on the optimized environment features at time t to obtain the real-time local semantic environment at time t; wherein, the real-time local semantic environment includes the point sequence and category of environmental elements; The update module is used to replace the historical local semantic environment corresponding to the vehicle position at time t in the historical global semantic environment with the real-time local semantic environment at time t.
9. An electronic device, characterized in that, include: Memory, used to store executable instructions; A processor, when executing executable instructions stored in the memory, implements the method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, It stores executable instructions for implementing the method according to any one of claims 1 to 7 when executed by a processor.
