Environment sensing method and roadside sensing system

Abstract

The invention provides an environment sensing method and a roadside sensing system. The method includes: obtaining, by an encoder, a 3D feature map representing a current state of an automatic driving environment based on a 2D image obtained by capturing the automatic driving environment in real time by a camera located on a roadside; obtaining a 3D reference feature map representing the historical state of the automatic driving environment based on the 3D reference data of the automatic driving environment through a multi-layer sensor; fusing the 3D feature map and the 3D reference feature map through a fusion device to obtain fusion information of the two feature maps; and obtaining differentiation information representing the difference of the current state of the automatic driving environment relative to the historical state based on the fusion information through a decoder.

Description

Technical Field

[0001] The present invention relates to an environment perception method based on a deep neural network and a roadside perception system. Background Art

[0002] The rapid development of intelligent transportation has put forward high requirements on environmental perception technology. In the prior art, foreground extraction (foreground segmentation) is usually used to obtain environmental results. However, this has high requirements for the reference image, and there is a problem that a single image may not be able to reliably describe the perceived environment. For example, in the case of stains on the camera or changes in lighting, a single image will not be able to accurately express the perceived environment. In addition, different lighting and weather changes will affect the information expression of a single image, making the single image almost unsuitable for reference. Summary of the invention

[0003] In view of the above-mentioned problems in the prior art, one aspect of the present invention provides a method for perceiving an autonomous driving environment, comprising: obtaining, through an encoder, a 3D feature map representing the current state of the autonomous driving environment based on a 2D image obtained by a camera located on the roadside in real time capturing the autonomous driving environment; obtaining, through a multilayer perceptron, a 3D reference feature map representing the historical state of the autonomous driving environment based on 3D reference data of the autonomous driving environment; fusing, through a fuser, the 3D feature map and the 3D reference feature map to obtain fusion information of the feature maps of the two; and obtaining, through a decoder, differentiated information representing the difference between the current state of the autonomous driving environment and the historical state based on the fusion information.

[0004] In one embodiment, the 3D reference data includes 3D reference data that implicitly expresses the autonomous driving environment and / or 3D reference data that explicitly expresses the autonomous driving environment.

[0005] In one embodiment, the 3D reference data for implicitly expressing the autonomous driving environment includes one or more of the following: SDF and NeRF; and wherein the 3D reference data for explicitly expressing the autonomous driving environment includes one or more of the following: 3D point cloud and mesh.

[0006] In one embodiment, the multilayer perceptron is pre-trained, and the pre-training includes: using the 3D reference data as training data and selecting a loss function based on an environmental perception task related to autonomous driving to enable the multilayer perceptron to update weight parameters through self-supervised learning, and storing the learned environmental information in the multilayer perceptron.

[0007] In one embodiment, the multilayer perceptron is continuously updated, and the continuous updating includes: obtaining new 3D reference data based on new 2D images of the autonomous driving environment obtained within a predetermined time period through a BEV encoder; and using the new 3D reference data as training parameters to enable the multilayer perceptron to update weight parameters through self-supervised learning.

[0008] In one embodiment, the method further comprises: obtaining, by a BEV encoder, the 3D reference data in the BEV space based on a multi-view 2D image of an autonomous driving environment.

[0009] In one embodiment, the method further includes: preprocessing the 3D reference data through a preprocessing module, and the preprocessing includes one or any multiple of the following: acquiring dense 3D reference data representing the historical state of the autonomous driving environment based on multiple sparse 3D reference data representing the historical state of the autonomous driving environment; acquiring fused 3D reference data representing the historical state of the autonomous driving environment based on multiple 3D reference data representing the historical state of the autonomous driving environment from a multimodal sensor; eliminating the influence of lighting changes and weather changes in the autonomous driving environment on the 3D reference data.

[0010] In one embodiment, the difference between the current state of the autonomous driving environment and the historical state includes: the difference between the foreground and / or background between the two states.

[0011] Another aspect of the present invention provides a roadside perception system, which includes: a camera, arranged on a roadside facility and configured to capture an autonomous driving environment in real time and output a 2D image; a deep neural network, configured to execute the method described above to obtain differentiated information representing the difference between a current state of the autonomous driving environment and a historical state; and a communication unit, configured to send the differentiated information to one or more of a vehicle, an edge cloud, and a cloud server.

[0012] Yet another aspect of the present invention provides a machine-readable storage medium storing executable instructions, which, when executed, enable one or more processors to perform the above-mentioned environment perception method.

[0013] The above presents a summary of major aspects of the invention in order to provide a basic understanding of such aspects and as a prelude to the detailed description that is presented later. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The technical solution of the present invention will become more clear from the following detailed description in conjunction with the accompanying drawings. It is understood that these drawings are only used for exemplary description and are not intended to limit the scope of protection of the present invention.

[0015] Figure 1 is a schematic block diagram of a roadside perception system according to an embodiment of the present invention.

[0016] Figure 2 Shows Figure 1 A way to implement a camera for a roadside perception system in a vehicle.

[0017] Figure 3 to Figure 5 is an environment perception method according to some embodiments of the present invention. DETAILED DESCRIPTION

[0018] The embodiment of the present invention provides a method for perceiving an autonomous driving environment and a roadside perception system capable of implementing the method. According to the embodiment of the present invention, 2D image data representing the current state of the autonomous driving environment obtained by real-time capture of the autonomous driving environment by a roadside camera is merged with 3D reference data representing the historical state of the environment obtained by offline processing, thereby providing a perception result of the autonomous driving environment, that is, the difference between the current state of the autonomous driving environment and the historical state.

[0019] According to an embodiment of the present invention, the 3D reference data used for the above comparison may include 3D reference data that expresses the environment implicitly and / or explicitly. For example, the implicit 3D reference data may include any one or more of the following: a sign distance function (SDF), and a neural radiation field (NeRF). The displayed 3D reference data may include any one or more of the following: a point cloud, a mesh, and a voxel.

[0020] According to an embodiment of the present invention, the 3D reference data is preprocessed to eliminate the influence of environmental changes, thereby improving the robustness of environmental perception results. Environmental changes include, for example, one or both of illumination changes and weather changes. Eliminating environmental changes is to retain 3D geometric information that is not related to environmental changes, such as depth, structure, and semantic information.

[0021] According to an embodiment of the present invention, the decoder of the deep neural network can output differentiation information representing the difference between the current state of the autonomous driving environment and the historical state. The difference includes the difference between the background and / or foreground between the two states.

[0022] This differentiated information can provide accurate environmental perception results for autonomous driving tasks, thereby helping to better complete autonomous driving tasks. Compared with the perception results based on classification that are usually used in the prior art, the differentiated information according to the embodiments of the present invention is advantageous because objects that have not been learned may appear in the environment, that is, the object does not belong to any of the multiple predetermined categories, and thus cannot be accurately included in the perception results. For example, when wild animals or road debris appear in the environment, the perception results based on classification may not accurately express such changes in the environment. The differentiated information according to the embodiments of the present invention can solve this problem well.

[0023] In addition, this differentiated information can also be integrated with the perception results based on classification. Since the two can be heterogeneous in perception tasks and data, the possibility of common cause failure of the two perceptions is very low. Therefore, integrating the perceptions of the two can provide stable perception results for perception tasks involving vehicle safety, thereby improving vehicle safety.

[0024] In addition, the perception results processed and obtained on the roadside are transmitted to the vehicle side, so that the perception results obtained by the vehicle side are more comprehensive. Moreover, this method can also increase the safety of the vehicle side.

[0025] The specific embodiments of the present invention are described below with reference to the accompanying drawings.

[0026] Figure 1 1 shows a roadside perception system 100 according to an embodiment of the present invention, which is arranged on the roadside infrastructure and is used to provide useful information for the automatic driving of the vehicle 200. Figure 1 As shown, the roadside perception system 100 includes: a camera 10 , a deep neural network (DNN: Deep Neural Network) 20 and a communication module 30 .

[0027] The camera 10 is arranged on the roadside facility, and is used to capture a 2D image of the autonomous driving environment of the vehicle 200 in real time and output 2D image data. That is, the camera 10 shoots in real time to capture the current state of the autonomous driving environment.

[0028] In one embodiment, the camera 10 may include multiple cameras for collecting autonomous driving environment information from different viewing angles and outputting 2D image data corresponding to different viewing angles, thereby obtaining 2D image data of multiple viewing angles. Figure 2 , the camera 10 includes four cameras 10A to 10D, which collect autonomous driving environment information from four directions, namely, east, west, south, and north (one application scenario of this example is a road intersection). In this way, the autonomous driving environment information from multiple perspectives can be obtained through these four cameras. It can be understood that although Figure 2It is shown that the camera 10 includes four cameras. According to an embodiment of the present invention, the camera 10 may also include a greater or lesser number of cameras. These cameras may be arranged on different road sections in an autonomous driving environment, and the shooting angle of each camera may be arranged to best capture traffic participants on the road section.

[0029] In an embodiment of the present invention, DNN20 may include a preprocessing module that preprocesses multimodal sensor data expressing the historical state of the autonomous driving environment to obtain 3D reference data representing the historical state of the autonomous driving environment. The multimodal sensors include, for example, laser radar and high-definition maps (HD Map: high-definition map). The historical state of the environment expressed by the 3D reference data obtained in this way has better robustness and accuracy than the historical state expressed by a single image, thereby being able to provide high-quality reference data.

[0030] In an embodiment of the present invention, DNN can be implemented as a network architecture with an encoder-decoder. In some embodiments, DNN may also include a BEV (Bird Eye View) encoder, a transformer, a multilayer perceptron (MLP: Multilayer Perceptron) and a fusion module. These encoders, decoders, transformers, multilayer perceptrons and fusion modules constitute a new DNN architecture according to an embodiment of the present invention, and an environmental perception method according to an embodiment of the present invention is implemented based on the new DNN architecture.

[0031] In the embodiment of the present invention, the 3D reference data can be understood as having prior information, which, as reference data, is not affected by environmental changes or sensor degradation, and can best express the historical state of the environment.

[0032] In the embodiment of the present invention, the historical state of the environment can be understood as the state of the environment in the past period of time, for example, in the past day, week, month, year or two years.

[0033] In an embodiment of the present invention, the difference information may be associated with the reference information. For example, when the reference information is 3D reference information and 3D feature comparison is performed, the difference information may be represented in the form of 3D voxels.

[0034] In an embodiment of the present invention, the differential information may include the uncertainty of the difference between the current state and the historical state of the environment. The uncertainty may be expressed in terms of probability.

[0035] Differentiation information can be expressed in a manner that is suitable for the vehicle's autonomous driving perception task. For example, in the case where the autonomous driving task involves semantic segmentation, differentiation information is expressed in pixel-wise information, and each pixel can contain attribute information such as object category. In the case where the autonomous driving task involves instance segmentation, differentiation information is expressed in cluster pixel-wise information, which also contains object ID information. In the case where the autonomous driving task involves target detection, differentiation information is expressed in a 2D bounding box, and the attribute information of the object corresponding to the bounding box can be attached to the center point of the 2D bounding box.

[0036] It is understandable that the perception tasks of autonomous driving mainly include target detection (e.g., 2D / 3D bounding box), semantic segmentation, and voxel regression (3D voxel). For different perception tasks, the above explicit representations can be used at the task level to express differentiated information that does not match the reference information.

[0037] In one embodiment, for each of the multiple perspectives, a corresponding differential information can be obtained using a method according to an embodiment of the present invention, thereby obtaining multiple differential information. In addition, the multiple differential information can be fused to obtain fused differential information. The fused differential information can be used for autonomous driving perception tasks. It can be understood that although an example of obtaining multiple differential information based on multiple perspectives is described, according to an embodiment of the present invention, multiple differential information can also be obtained based on other information, without limitation thereto.

[0038] It is worth noting that the differentiated information can serve upstream tasks related to the planning and control of autonomous driving, such as target perception, lane line detection, etc. The fusion and post-processing of differentiated information can be adjusted accordingly according to different planning and control strategies. For example, the fused differentiated information is used as new upstream information to serve new autonomous driving tasks. For example, when it comes to new autonomous driving tasks with higher safety requirements, the differentiated information can be used to assist in obtaining grid information in the autonomous driving scene, such as occupancy information or non-free information of one or more grids.

[0039] The communication module 30 can be connected to the vehicle 200 in a wired and / or wireless manner. The communication module 30 can also be connected to the edge cloud (not shown) or the cloud server (not shown) in a wired and / or wireless manner. In this way, the differentiated information obtained on the roadside can be transmitted to one or more of the vehicle 200, the edge cloud and the cloud server via the communication module 30.

[0040] Next, combine Figure 3 to Figure 5To introduce the environmental perception method according to the embodiment of the present invention. These methods can be implemented with the help of the above-mentioned DNN processor 20 or the above-mentioned roadside perception system 100, so the above description of the DNN processor 20 and the roadside perception system 100 also applies here. It should be understood that the operations (steps) involved in the following methods do not have to be performed in the exact order described. Instead, multiple operations can be handled in different orders or simultaneously, and operations can be added or omitted.

[0041] Figure 3 An environment perception method 300 according to an embodiment of the present invention is shown. In this embodiment, through the above-mentioned novel DNN architecture, the difference between the current state of the environment and the historical state is obtained based on the 2D image data representing the current state of the autonomous driving environment and the 3D reference data that explicitly represents the historical state of the environment.

[0042] See also Figure 3 In block 310, a camera encoder (CAMERA ENCODER) obtains a 3D feature map (3D MAP) based on a 2D image (2D IMG) of the autonomous driving environment captured in real time by the camera 10. feature ). Here, the camera encoder can extract 2D features from the 2D image data to obtain a 2D feature map, and then project the 2D feature map into the 3D space to obtain the 3D feature map. Projecting the 2D feature map into the 3D space can be achieved, for example, by means of a geometric constraint relationship of pixel coordinates and / or a related machine learning model (e.g., transformer). It should be understood that the 3D feature map represents the current real-time state of the autonomous driving environment.

[0043] In box 320, the pre-processing module (PRE-PROCESSING MODULE) pre-processes the multiple point clouds that will explicitly express the autonomous driving environment to obtain 3D reference data (3D REF). The pre-processing may include one or more of the following: 1) matching the multiple point cloud data that explicitly express the autonomous driving environment to obtain 3D reference data (3D REF), wherein the multiple point cloud data may be sparse, and through the matching in box 320, dense 3D reference data can be obtained, so that the reference state of the autonomous driving environment can be more completely and accurately expressed; 2) pre-processing the multiple point cloud data to eliminate the influence of the illumination change and weather change of the autonomous driving environment on the 3D reference data, that is, obtaining a 3D reference feature map that is not affected by the illumination change and weather change of the environment, which can greatly improve the accuracy and robustness of environmental perception, because the background conditions obtained under different illumination may be very different; 3) fusing the 3D point clouds obtained by the multi-modal sensor, which can overcome the limitations of the single-modal sensor on the application scenario, illumination and weather.

[0044] In block 330, the BEV encoder (BEV ENCODER) obtains a 3D reference feature map (3D REF MAP ) in the BEV space based on the 3D reference data output by the pre-processing module. feature ). It should be understood that the 3D reference feature map represents the historical state of the autonomous driving environment with background prior information, which is used as a reference in the subsequent fusion process.

[0045] In block 340, the fusion module (FUSION MODULE) fuses the 3D feature map output by the camera encoder with the 3D reference feature map output by the BEV encoder to obtain fusion information. Here, fusion can be achieved by cross-attention processing (Cross-Attention) and / or concatenation processing (concatenation).

[0046] In one embodiment, the fusion can be implemented as an independent task. The fusion can also be combined with other algorithms to ensure the heterogeneity of the tasks. For example, a predetermined loss function (e.g., KLloss (Kullback-Leibler divergence loss) for Feature map / Space) is used to amplify the difference between the two feature maps (i.e., the 3D feature map and the 3D reference feature map).

[0047] In box 350, the decoder (DECODER) obtains difference information (Info_Diff) based on the fusion information output by the fusion module, which characterizes the difference between the current state of the autonomous driving environment and the historical state. The difference should be understood as the difference between the foreground and / or background of the two states (i.e., the change or difference between the two states). For example, the difference may include a tree that disappears in the background or a newly appearing spilled object. The difference may also include a newly appearing vehicle or pedestrian in the foreground.

[0048] Additionally, the communication module 30 may transmit the differentiation information to one or more of the vehicle 200 , the edge cloud, and the cloud service.

[0049] After the differentiated information is sent to the vehicle, it can be integrated with other environmental perception results implemented on the vehicle (for example, the visibility grid map of the autonomous driving environment) to obtain a fused perception result. The fused perception result can be applied to multiple functions of vehicle driving assistance.

[0050] In the example where the differentiated information is sent to the edge server, it can be sent to the vehicle end based on the vehicle request when the vehicle approaches the edge server.

[0051] In the example where the differentiated information is sent to the cloud server, the differentiated information can be sent to the vehicle end at a regular interval (every morning before the vehicle departs or every evening after the vehicle completes a day's driving mission).

[0052] Figure 4 An environment perception method 400 according to another embodiment of the present invention is shown.

[0053] See also Figure 4 In block 410, similar to block 310, the encoder obtains a 3D feature map (3DMAP) based on a 2D image of the autonomous driving environment captured in real time by the camera 10. feature ). The 3D feature map represents the current real-time state of the autonomous driving environment. In one embodiment, the encoder is implemented as a BEV encoder (BEV ENCODER), and the 3D feature map is a 3D feature map in the BEV space.

[0054] At block 420, 3D reference data (3D REF) implicitly expressing the autonomous driving environment is input into a multi-layer perceptron (MLP), and a 3D reference feature map (3D REF MAP) is output from the MLP. feature ). It should be understood that the 3D reference feature map represents the historical state of the autonomous driving environment with background prior information, which is used as a reference in the subsequent fusion process.

[0055] The implicit expression of the autonomous driving environment may include one or more of the following: a sign distance function (SDF), and a neural radiance field (NeRF), for example, a language embedded radiance field (LERF).

[0056] In one embodiment, the MLP is pre-trained. For example, the 3D reference data is used as training data to train the MLP, and a loss function is selected according to an environment perception task related to autonomous driving, so that the MLP iteratively updates weight parameters through self-supervised learning, and stores the learned environment information (e.g., environment information implicitly represented by SDF or NeRF) in the MLP, and then outputs a 3D reference feature map through position information input (e.g., specific position and direction).

[0057] In one embodiment, the MLP is continuously updated. For example, see Figure 5In boxes 510 to 540, each 2D image (2D IMG) in a plurality of 2D images (e.g., multi-view 2D images) obtained within a predetermined time period is input into a camera encoder (CAMERA ENCODER) to extract image features. In box 550, the extracted image features are input into a BEV encoder (BEVENCODER) to obtain new 3D reference data. Next, the new 3D reference data is used as a training parameter so that the MLP updates the weight parameters through self-supervised learning. In this way, continuous updating of the MLP is achieved. The update can be performed periodically so that the MLP becomes more sensitive to the difference between the current state of the environment and the historical state.

[0058] It should be understood that the 3D reference data input in 420 may be pre-processed as described in reference block 320 , so that the 3D reference data has the advantages of accuracy and robustness.

[0059] return Figure 4 In block 430, a fusion module (FUSION MODULE) fuses the 3D feature map output by the BEV encoder with the 3D reference feature map output by the MLP to output fusion information. The fusion can be achieved by cross-attention processing (Cross-Attention) and / or concatenation processing (concatenation).

[0060] In one embodiment, the fusion can be implemented as an independent task. The fusion can also be combined with other algorithms to ensure the heterogeneity of the tasks. For example, a predetermined loss function is used to amplify the difference between the two feature maps (i.e., the 3D feature map and the 3D reference feature map).

[0061] In block 440, the decoder (DECODER) decodes the fusion information output by the fusion module to obtain difference information (Info_Diff), which represents the difference between the current state of the autonomous driving environment and the historical state. The difference should be understood as the difference between the foreground and / or background of the two states (i.e., the change or difference between the two states).

[0062] Additionally, the communication module 30 may transmit the differentiation information to one or more of the vehicle 200 , the edge cloud, and the cloud service.

[0063] After the differentiated information is sent to the vehicle, it can be integrated with other environmental perception results implemented on the vehicle (for example, the visibility grid map of the autonomous driving environment) to obtain a fused perception result. The fused perception result can be applied to multiple functions of vehicle driving assistance.

[0064] In the example where the differentiated information is sent to the edge server, it can be sent to the vehicle end based on the vehicle request when the vehicle approaches the edge server.

[0065] In the example where the differentiated information is sent to the cloud server, the differentiated information can be sent to the vehicle end at a regular interval (every morning before the vehicle departs or every evening after the vehicle completes a day's driving mission).

[0066] According to another embodiment of the present invention, 3D reference data that implicitly expresses the autonomous driving environment and 3D reference data that explicitly expresses the autonomous driving environment may be converted into each other (i.e., the displayed 3D reference data may be converted into implicit reference data, or the implicit 3D reference data may be converted into displayed 3D reference data), so that 3D reference data that is more suitable for the current application scenario may be used as training data for the MLP.

[0067] The present invention also provides a machine-readable storage medium storing executable instructions, which, when executed, enable one or more processors to perform the above-mentioned environment perception method.

[0068] It is understood that the processor can be implemented using electronic hardware, computer software or any combination thereof. Whether these processors are implemented as hardware or software will depend on specific application and the overall design constraints imposed on the system. As an example, the processor provided in the present invention, any part of the processor or any combination of the processor can be implemented as a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA), a programmable logic device (PLD), a state machine, a gate logic, a discrete hardware circuit and other suitable processing components configured to perform the various functions described in the present invention. The function of the processor provided in the present invention, any part of the processor or any combination of the processor can be implemented as software performed by a microprocessor, a microcontroller, a DSP or other suitable platforms.

[0069] Software can be broadly considered to represent instructions, instruction sets, codes, code segments, program codes, programs, subroutines, software modules, applications, software applications, software packages, routines, subroutines, objects, running threads, processes, functions, etc. Software can reside in a computer-readable medium. A computer-readable medium can include, for example, a memory, which can be, for example, a magnetic storage device (e.g., a hard disk, a floppy disk, a magnetic stripe), an optical disk, a smart card, a flash memory device, a random access memory (RAM), a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), a register, or a removable disk. Although the memory is shown as being separated from the processor in the various aspects provided by the present invention, the memory can also be located inside the processor (e.g., a cache or a register).

[0070] The above description is provided to enable any person skilled in the art to implement the various aspects described herein. The claims are not intended to be limited to the aspects shown herein. All structural and functional equivalents of the elements of the various aspects described in the present invention that are known or to be known to those skilled in the art will be covered by the claims of the present invention and are within the scope of protection of the present invention.

Claims

1. A method for perceiving an autonomous driving environment, comprising: Obtaining, by an encoder, a 3D feature map representing a current state of the autonomous driving environment based on a 2D image obtained by capturing the autonomous driving environment in real time by a camera located on the roadside; Obtaining, by a multi-layer perceptron, a 3D reference feature map representing a historical state of the autonomous driving environment based on the 3D reference data of the autonomous driving environment; The 3D feature map and the 3D reference feature map are fused by a fuser to obtain fusion information of the feature maps of the two. as well as Through the decoder, differentiated information representing the difference between the current state of the autonomous driving environment and the historical state is obtained based on the fusion information.

2. The method of claim 1, wherein: The 3D reference data includes 3D reference data that implicitly expresses the autonomous driving environment and / or 3D reference data that explicitly expresses the autonomous driving environment.

3. The method of claim 2, wherein: The 3D reference data implicitly expressing the autonomous driving environment includes one or more of the following: SDF and NeRF; and The 3D reference data for explicitly expressing the autonomous driving environment includes one or more of the following: 3D point cloud and mesh.

4. The method according to any one of claims 1 to 3, wherein: The multilayer perceptron is pre-trained, and the pre-training includes: The 3D reference data is used as training data and a loss function is selected according to an environment perception task related to autonomous driving to enable the multilayer perceptron to update weight parameters through self-supervised learning, and the learned environment information is stored in the multilayer perceptron.

5. The method according to any one of claims 1 to 4, wherein: The multilayer perceptron is continuously updated, and the continuous updating includes: obtaining, by the BEV encoder, new 3D reference data based on new 2D images of the autonomous driving environment obtained within a predetermined time period; and The new 3D reference data is used as training parameters to enable the multilayer perceptron to update the weight parameters through self-supervised learning.

6. The method according to any one of claims 1 to 5, further comprising: The 3D reference data in the BEV space is obtained through the BEV encoder based on the multi-view 2D images of the autonomous driving environment.

7. The method according to any one of claims 1 to 6, further comprising: The 3D reference data is preprocessed by a preprocessing module, and the preprocessing includes one or more of the following: Acquire dense 3D reference data representing the historical state of the autonomous driving environment based on a plurality of sparse 3D reference data representing the historical state of the autonomous driving environment; Obtaining fused 3D reference data representing the historical state of the autonomous driving environment based on a plurality of 3D reference data representing the historical state of the autonomous driving environment from a multimodal sensor; Eliminate the impact of lighting and weather changes in the autonomous driving environment on 3D reference data.

8. The method according to any one of claims 1 to 6, wherein: The difference between the current state of the autonomous driving environment and the historical state includes: the difference between the foreground and / or background between the two states.

9. A roadside perception system, comprising: A camera, arranged on a roadside facility and configured to capture the autonomous driving environment in real time and output a 2D image; A deep neural network configured to execute the method according to any one of claims 1 to 8 to obtain differentiated information representing a difference between a current state of the autonomous driving environment and a historical state; as well as A communication unit is configured to send the differentiated information to one or more of a vehicle, an edge cloud, and a cloud server.

10. A machine-readable storage medium storing executable instructions, which, when executed, cause one or more processors to perform the environment perception method according to any one of claims 1 to 8.

Images