Driving environment sensing method and device
By integrating prior and current BEV characteristics in the BEV network, the problem of instability of perceived results in complex urban market scenarios is solved, and the integrity and stability of driving environment perception is improved.
Patent Information
- Application Number
- CN202310749389.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-21
- Publication Date
- 2024-12-24
AI Technical Summary
The existing Bird Eye's View (BEV) network has unstable perception results in complex urban market scenarios and cannot fully output perceptual results, especially when facing roundabouts, super large intersections and left-turn blocking intersections.
By obtaining the vehicle's environmental data and location information, sending location information to the server and receiving prior BEV features, combining environmental data to generate current BEV features, and performing feature fusion to obtain fused BEV features, thereby improving the integrity and stability of the perceived results of the driving environment sex.
The prior BEV features are used to reduce the impact of complex scenarios and occlusion on the perceived results, and improve the integrity and stability of the driving environment perception.
Smart Images
Figure CN119176140A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of vehicle technology, and in particular to a driving environment perception method and device. Background Art
[0002] At present, the Bird's Eye's View (BEV) network has gradually entered the algorithm architecture of multiple intelligent driving products. Compared with the detection and segmentation scheme of traditional vision, the BEV network has shown its powerful perception and generalization capabilities. However, in the face of complex urban scenes, such as roundabouts, super-large intersections, and left-turn blocked intersections, the perception of the BEV network alone is limited, that is, it cannot output the perception results sufficiently complete and stable. Summary of the invention
[0003] In view of this, an embodiment of the present application provides a driving environment perception method and device for solving the problem that the perception results of the existing BEV network are limited and unstable.
[0004] In a first aspect, an embodiment of the present application provides a driving environment perception method, comprising:
[0005] Acquiring environmental data of a vehicle's driving environment and location information of the vehicle;
[0006] Sending the location information to a server;
[0007] receiving a first BEV feature corresponding to the location information sent by the server;
[0008] generating a second BEV characteristic based on the environmental data;
[0009] Performing feature fusion on the first BEV feature and the second BEV feature to obtain a fused BEV feature;
[0010] A perception result of the driving environment is obtained based on the fused BEV feature.
[0011] As an optional implementation of the embodiment of the present application, after generating the second BEV feature according to the environmental data, the method further includes:
[0012] Sending update indication information carrying the location information and the fused BEV feature to the server;
[0013] Among them, it carries the location information and the update indication information of the fused BEV feature.
[0014] As an optional implementation of the embodiment of the present application, the environmental data includes at least two types of sensor data acquired by at least two sensors; and generating the second BEV feature according to the environmental data includes:
[0015] Extracting BEV features from the at least two sensor data respectively to obtain BEV features corresponding to the at least two sensor data respectively;
[0016] Perform spatial feature fusion on the BEV features respectively corresponding to the at least two sensor data to generate the second BEV feature.
[0017] As an optional implementation of the embodiment of the present application, generating the second BEV feature according to the environmental data includes:
[0018] generating a current BEV characteristic based on the environmental data;
[0019] Acquire at least one historical BEV feature; the historical BEV feature is a BEV feature generated according to the historical environmental data corresponding to the location information;
[0020] Performing time series feature fusion on the current BEV feature and the at least one historical BEV feature to generate the second BEV feature.
[0021] As an optional implementation of the embodiment of the present application,
[0022] The environmental data includes: at least one of image data, laser radar data, millimeter wave radar data, and distance sensor data;
[0023] The perception result includes: at least one of road structural elements, traffic lights, and road topological relationships.
[0024] In a second aspect, an embodiment of the present application provides a driving environment perception method, comprising:
[0025] Receive location information sent by the vehicle;
[0026] Extracting a first bird's-eye view BEV feature corresponding to the location information from a BEV feature library;
[0027] The first BEV feature is sent to the vehicle so that the vehicle fuses the first BEV feature with a second BEV feature generated according to environmental data of the driving environment of the vehicle to obtain a fused BEV feature, and obtains a perception result of the driving environment based on the fused BEV feature.
[0028] As an optional implementation of the embodiment of the present application, the method further includes:
[0029] receiving update indication information sent by the vehicle and carrying the location information and the fused BEV feature;
[0030] In response to the update indication information, the BEV feature corresponding to the location information in the BEV feature library is updated according to the fused BEV feature.
[0031] As an optional implementation of the embodiment of the present application, before extracting the first BEV feature corresponding to the location information from the BEV feature library, the method further includes:
[0032] Environmental data corresponding to multiple location information obtained based on shadow mode;
[0033] Generate BEV features corresponding to each position information according to the environmental data corresponding to each position information;
[0034] The BEV features corresponding to each piece of position information are saved in the BEV feature library.
[0035] In a third aspect, an embodiment of the present application provides a vehicle, including:
[0036] An information acquisition module, used to obtain environmental data of the vehicle's driving environment and location information of the vehicle;
[0037] a communication module, configured to send the location information to a server, and receive a first bird's-eye view BEV feature corresponding to the location information sent by the server;
[0038] a feature generation module, configured to generate a second BEV feature according to the environmental data;
[0039] A feature fusion module, used for fusing the first BEV feature and the second BEV feature to obtain a fused BEV feature;
[0040] An environment perception module is used to obtain a perception result of the driving environment based on the fused BEV features.
[0041] As an optional implementation of the embodiment of the present application, the communication module is also used to send update indication information carrying the location information and the fused BEV feature to the server after generating the second BEV feature according to the environmental data; wherein, the update indication information carrying the location information and the fused BEV feature.
[0042] As an optional implementation of the embodiment of the present application, the environmental data includes at least two sensor data obtained by at least two sensors; the feature generation module is specifically used to extract BEV features from the at least two sensor data respectively, obtain BEV features corresponding to the at least two sensor data respectively, and perform spatial feature fusion on the BEV features corresponding to the at least two sensor data respectively to generate the second BEV feature.
[0043] As an optional implementation of the embodiment of the present application, the feature generation module is specifically used to generate a current BEV feature based on the environmental data; obtain at least one historical BEV feature; the historical BEV feature is to generate a BEV feature based on the historical environmental data corresponding to the location information, and perform time series feature fusion on the current BEV feature and the at least one historical BEV feature to generate the second BEV feature.
[0044] As an optional implementation of the embodiment of the present application,
[0045] The environmental data includes: at least one of image data, laser radar data, millimeter wave radar data, and distance sensor data;
[0046] The perception result includes: at least one of road structural elements, traffic lights, and road topological relationships.
[0047] In a fourth aspect, an embodiment of the present application provides a server, including:
[0048] A receiving module, used for receiving location information sent by the vehicle;
[0049] A processing module, configured to extract a first bird's-eye view BEV feature corresponding to the position information from a BEV feature library;
[0050] A sending module is used to send the first BEV feature to the vehicle, so that the vehicle fuses the first BEV feature with a second BEV feature generated according to environmental data of the driving environment of the vehicle to obtain a fused BEV feature, and obtain a perception result of the driving environment based on the fused BEV feature.
[0051] As an optional implementation of the embodiment of the present application,
[0052] The receiving module is further used to receive update indication information sent by the vehicle and carrying the location information and the fused BEV feature;
[0053] The processing module is further configured to update the BEV feature corresponding to the location information in the BEV feature library according to the fused BEV feature in response to the update indication information.
[0054] As an optional implementation of the embodiment of the present application, the processing unit is also used to obtain environmental data corresponding to multiple location information based on the shadow mode before extracting the first BEV feature corresponding to the location information from the BEV feature library; generate BEV features corresponding to each location information according to the environmental data corresponding to each location information; and save the BEV features corresponding to each location information into the BEV feature library.
[0055] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising: a memory and a processor, wherein the memory is used to store a computer program; and the processor is used to enable the electronic device to implement the driving environment perception method described in any one of the above embodiments when executing the computer program.
[0056] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a computing device, the computing device implements the driving environment perception method described in any one of the above embodiments.
[0057] The driving environment perception method provided in the embodiment of the present application will send the location information of the vehicle to the server after acquiring the environmental data of the driving environment of the vehicle and the location information of the vehicle, and receive the first BEV feature corresponding to the location information sent by the server, and generate the second BEV feature according to the environmental data of the driving environment, and then fuse the first BEV feature and the second BEV feature to obtain the fused BEV feature, and obtain the perception result of the driving environment based on the fused BEV feature. Since the embodiment of the present application can use the location information to obtain the prior BEV feature from the server, and fuse the prior BEV feature and the current BEV feature to perceive the driving environment according to the fused feature, the embodiment of the present application can use the prior BEV feature to reduce the influence of complex scenes, occlusion, etc. on the perception result of the driving environment, thereby improving the integrity and stability of the perception result of the driving environment. BRIEF DESCRIPTION OF THE DRAWINGS
[0058] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0059] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings required in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0060] Figure 1 A scenario architecture diagram of the driving environment perception method provided in an embodiment of the present application;
[0061] Figure 2 One of the interactive flow charts of the driving environment perception method provided in the embodiment of the present application;
[0062] Figure 3One of the structural schematic diagrams of a vehicle for implementing a driving environment perception method provided in an embodiment of the present application;
[0063] Figure 4 The second interactive flow chart of the driving environment perception method provided in the embodiment of the present application;
[0064] Figure 5 A second structural schematic diagram of a vehicle for implementing a driving environment perception method provided in an embodiment of the present application;
[0065] Figure 6 The third interactive flow chart of the driving environment perception method provided in the embodiment of the present application;
[0066] Figure 7 A third structural schematic diagram of a vehicle for implementing a driving environment perception method provided in an embodiment of the present application;
[0067] Figure 8 A fourth structural schematic diagram of a vehicle for implementing a driving environment perception method provided in an embodiment of the present application;
[0068] Fig. 9 A schematic diagram of a device for implementing a driving environment perception method provided in an embodiment of the present application;
[0069] Fig.10 A schematic diagram of the structure of a vehicle provided in an embodiment of the present application;
[0070] Fig.11 A schematic diagram of the structure of a server provided in an embodiment of the present application;
[0071] Fig.12 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0072] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the scheme of the present application will be further described below. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.
[0073] In the following description, many specific details are set forth to facilitate a full understanding of the present application, but the present application may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only part of the embodiments of the present application, rather than all of the embodiments.
[0074] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way. In addition, in the description of the embodiments of the present application, unless otherwise specified, the meaning of "multiple" refers to two or more.
[0075] It should be noted that, in this article, the term "comprises" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.
[0076] The following first describes the scene structure of the driving environment perception method provided in an embodiment of the present application.
[0077] Reference Figure 1 As shown, the driving environment perception method provided in the embodiment of the present application includes: a vehicle 100 and a server 200. A wireless communication link is established between the vehicle 100 and the server 200, and data can be transmitted between the vehicle 100 and the server 200 through the established wireless communication link.
[0078] The vehicle 100 is equipped with one or more of the sensors such as an image acquisition device, a distance sensor, a laser radar, and a millimeter-wave radar. The environmental data of the driving environment can be obtained through the image acquisition device, the distance sensor, the laser radar, the millimeter-wave radar, and the like, and the BEV features can be generated according to the environmental data obtained. On the other hand, the vehicle 100 is also equipped with a positioning device, which can obtain the location information of the vehicle in real time. After the vehicle obtains the location information, it can send the obtained location information to the server 200 through a wireless communication link, and receive the prior BEV features corresponding to the location information sent by the server 200. After the vehicle 100 obtains the prior BEV features sent by the server 200, it can also fuse the prior BEV features with the BEV features generated according to the environmental data, and perceive the driving environment according to the fused features.
[0079] The server 200 is equipped with a BEV feature library, in which a plurality of BEV features corresponding to position information are stored. When receiving the position information sent by the vehicle 100, the server 200 can extract the corresponding BEV features from the BEV feature library according to the position information sent by the vehicle 100, and send the extracted BEV features to the vehicle 100. The BEV features in the BEV feature library can be BEV features generated by the server 200 according to the environmental data uploaded by the vehicle, or can be BEV features directly uploaded by the vehicle, and the server 200 can also update the BEV features in the BEV feature library according to the BEV features uploaded by the vehicle.
[0080] Based on the above content, the embodiment of the present application provides a driving environment perception method, referring to Figure 2 As shown, the driving environment perception method includes the following steps S201 to S207:
[0081] S201. The vehicle obtains environmental data of the vehicle's driving environment and location information of the vehicle.
[0082] In some embodiments, the environmental data of the driving environment may include at least one of image data, lidar data, millimeter wave radar data, and distance sensor data.
[0083] In some embodiments, the vehicle's location information can be obtained by a vehicle-mounted positioning system such as a vehicle-mounted global positioning system (GPS) and a vehicle-mounted Beidou positioning system.
[0084] S202: The vehicle sends the location information to the server.
[0085] Correspondingly, the server receives the location information sent by the vehicle.
[0086] In some embodiments, a wireless communication link may be established between the vehicle and the server, and the vehicle may send the location information to the server via the wireless communication link established between the vehicle and the server.
[0087] S203: The server extracts a first BEV feature corresponding to the location information from a BEV feature library.
[0088] In some embodiments, the BEV feature library may store the location information and the BEV features corresponding to the location information in the form of a correspondence table. When the location information sent by the vehicle is received, the correspondence table may be searched according to the location information to obtain the BEV features corresponding to the location information.
[0089] It should be noted that the BEV features corresponding to the location information in the embodiment of the present application may include only one BEV feature or multiple BEV features. That is, when the server obtains multiple BEV features for the same location information, the server may save all the multiple features as BEV features corresponding to the location information, and send the multiple BEV features to the vehicle when receiving the location information sent by the vehicle. The server may also fuse the multiple BEV features and save the fused BEV features as the BEV features corresponding to the location information, and send the fused BEV features to the vehicle when receiving the location information sent by the vehicle.
[0090] Exemplarily, the corresponding relationship table storing the location information and the BEV features corresponding to the location information may be shown in Table 1 below:
[0091] Table 1
[0092]
[0093] S204: The server sends the first BEV feature to the vehicle.
[0094] Correspondingly, the vehicle receives the first BEV feature corresponding to the location information sent by the server.
[0095] Likewise, the server may send the first BEV feature to the vehicle via a wireless communication link established between the server and the vehicle.
[0096] S205: The vehicle generates a second BEV feature according to the environmental data.
[0097] The implementation method of generating BEV features according to environmental data is not limited in the embodiments of the present application. Specifically, any corresponding BEV feature extraction algorithm can be used to obtain BEV features according to the specific environmental data. For example, when the environmental data is image data, BEV Former (a Transformer-based network structure, the main feature of which is to apply the self-attention mechanism on the BEV network to extract BEV features) and LSS (Lift, Splat, Shoot) algorithm (a method of generating 3D features from two-dimensional image features, and then mapping the 3D features to obtain a BEV feature map) can be used.
[0098] It should be noted that in the embodiment of the present application, the order of generating the second BEV feature according to the environmental data (S205) and obtaining the first BEV feature from the server (S202~S204) is limited. The first BEV feature can be obtained from the server first, and then the second BEV feature can be generated according to the environmental data. The second BEV feature can be generated according to the environmental data first, and then the first BEV feature can be obtained from the server. The second BEV feature can also be generated according to the environmental data and the first BEV feature can be obtained from the server at the same time.
[0099] S206: The vehicle performs feature fusion on the first BEV feature and the second BEV feature to obtain a fused BEV feature.
[0100] In the embodiment of the present application, the feature fusion algorithm used by the vehicle when fusing the first BEV feature and the second BEV feature is not limited, and the feature fusion algorithm is capable of fusing the first BEV feature and the second BEV feature and obtaining the fused BEV feature. Exemplarily, the feature fusion algorithm may be a Bayesian-based feature fusion algorithm, a sparse representation-based feature fusion algorithm, a deep learning-based feature fusion algorithm, etc.
[0101] S207: The vehicle obtains a perception result of the driving environment based on the fused BEV feature.
[0102] In some embodiments, the perception result of the driving environment may include: at least one of: road structure elements, traffic lights, and road topological relationships. Among them, the road structure elements may include: lane lines, curbs, stop lines, zebra crossings, etc.
[0103] Reference Figure 3 As shown, in some embodiments, the driving environment perception device in the vehicle that implements the driving environment perception method provided in the above embodiment may include: a feature extraction module 31, a feature fusion module 32 and a perception module 33. Among them, the input of the feature extraction module 31 is the environmental data of the driving environment of the vehicle, which is used to generate and output the second BEV feature according to the environmental data of the driving environment of the vehicle. The input of the feature fusion module 32 is the second BEV feature output by the feature extraction module 31 and the first BEV feature obtained from the server, which is used to perform feature fusion on the second BEV feature and the first BEV feature to obtain the fused BEV feature and output it. The input of the perception module 33 is the fused BEV feature output by the feature fusion module 32, which is used to obtain and output the perception result of the driving environment based on the fused BEV feature. Among them, the perception module 33 can process a variety of perception tasks based on the fused BEV feature, such as: road structure element detection, traffic light perception, topological relationship prediction, etc.
[0104] The driving environment perception method provided in the embodiment of the present application will send the location information of the vehicle to the server after acquiring the environmental data of the driving environment of the vehicle and the location information of the vehicle, and receive the first BEV feature corresponding to the location information sent by the server, and generate the second BEV feature according to the environmental data of the driving environment, and then fuse the first BEV feature and the second BEV feature to obtain the fused BEV feature, and obtain the perception result of the driving environment based on the fused BEV feature. Since the embodiment of the present application can use the location information to obtain the prior BEV feature from the server, and fuse the prior BEV feature and the current BEV feature to perceive the driving environment according to the fused feature, the embodiment of the present application can use the prior BEV feature to reduce the influence of complex scenes, occlusion, etc. on the perception result of the driving environment, thereby improving the integrity and stability of the perception result of the driving environment.
[0105] In some embodiments, the present application provides another driving environment perception method, referring to Figure 4 As shown, the driving environment perception method includes the following steps S401 to S410:
[0106] S401. The vehicle obtains environmental data of the vehicle's driving environment and location information of the vehicle.
[0107] The environmental data includes at least two types of sensor data acquired by at least two types of sensors.
[0108] The implementation method for the vehicle to obtain environmental data and location information of the driving environment can refer to the implementation method for obtaining environmental data and location information of the driving environment in the above step S201. To avoid redundancy, it will not be described in detail here.
[0109] S402: The vehicle extracts BEV features from the at least two sensor data respectively to obtain BEV features corresponding to the at least two sensor data respectively.
[0110] S403: The vehicle performs spatial feature fusion on the BEV features corresponding to the at least two sensor data to generate the second BEV feature.
[0111] Since the above-mentioned embodiment can respectively perform BEV feature extraction on the at least two sensor data to obtain the BEV features corresponding to the at least two sensor data, and perform spatial feature fusion on the BEV features corresponding to the at least two sensor data to generate the second BEV feature, the driving environment perception method provided in the embodiment of the present application can enable the vehicle to obtain the second BEV feature based on the environmental data more comprehensively and stably, thereby further improving the integrity and stability of the perception results of the driving environment.
[0112] S404: The vehicle sends the location information to the server.
[0113] Correspondingly, the server receives the location information sent by the vehicle.
[0114] S405. The server extracts a first BEV feature corresponding to the location information from a BEV feature library.
[0115] S406: The server sends the first BEV feature to the vehicle.
[0116] Correspondingly, the vehicle receives the first BEV feature corresponding to the location information sent by the server.
[0117] S407: The vehicle performs feature fusion on the first BEV feature and the second BEV feature to obtain a fused BEV feature.
[0118] S408: The vehicle obtains a perception result of the driving environment based on the fused BEV features.
[0119] S409: The vehicle sends update indication information carrying the location information and the fused BEV features to the server.
[0120] Correspondingly, the server receives update indication information sent by the vehicle, which carries the location information and the fused BEV feature.
[0121] Among them, it carries the location information and the update indication information of the fused BEV feature.
[0122] S410. In response to the update indication information, the server updates the BEV feature corresponding to the location information in the BEV feature library according to the fused BEV feature.
[0123] In some embodiments, the server updates the BEV feature corresponding to the location information in the BEV feature library according to the fused BEV feature, including: replacing the BEV feature corresponding to the location information in the BEV feature library with the fused BEV feature.
[0124] In some embodiments, the server updates the BEV feature corresponding to the location information in the BEV feature library based on the fused BEV feature, including: fusing the BEV feature corresponding to the location information in the BEV feature library and the fused BEV feature, and updating the BEV feature obtained by fusing the BEV feature corresponding to the location information in the BEV feature library and the fused BEV feature to the BEV feature corresponding to the location information.
[0125] The above-mentioned embodiment can also send the fused BEV feature to the server after the vehicle fuses the first BEV feature and the second BEV feature to obtain the fused BEV feature, so that the server updates the BEV feature corresponding to the location information in the database according to the fused BEV feature sent by the vehicle. Therefore, the above-mentioned embodiment can update the BEV feature corresponding to the location information in time to avoid changes in the driving environment corresponding to the location information, and the BEV features stored in the BEV feature library are updated accordingly.
[0126] Reference Figure 5 As shown, in some embodiments, the driving environment perception device in the vehicle that implements the driving environment perception method provided in the above embodiment may include: a feature extraction module 51, a feature fusion module 52 and a perception module 53. Among them, the feature extraction module 51 includes: multiple feature extraction units 511 and a spatial fusion unit 512. The multiple feature extraction units 511 are respectively used to generate corresponding BEV features according to a type of sensor data. The spatial fusion unit 512 is used to perform spatial fusion on the BEV features output by the multiple feature extraction units 511 to obtain a second BEV feature. The feature fusion module 52 and the perception module 53 are respectively connected to Figure 3 The function of the feature fusion module 32 and the perception module 33 in the illustrated embodiment is the same and will not be described in detail here.
[0127] In some embodiments, the present application provides another driving environment perception method, referring to Figure 6 As shown, the driving environment perception method includes the following steps S601 to S611:
[0128] S601: The vehicle obtains environmental data of the vehicle's driving environment and location information of the vehicle.
[0129] The implementation method for the vehicle to obtain environmental data and location information of the driving environment can refer to the implementation method for obtaining environmental data and location information of the driving environment in the above step S201. To avoid redundancy, it will not be described in detail here.
[0130] S602: The vehicle generates current BEV characteristics according to the environmental data.
[0131] S603: The vehicle obtains at least one historical BEV feature.
[0132] The historical BEV feature is a BEV feature generated according to the historical environment data corresponding to the location information.
[0133] In some embodiments, the historical BEV feature may be a BE feature generated based on environmental data of the driving environment collected when the vehicle previously passed the location corresponding to the location information, and based on the environmental data of the driving environment.
[0134] S604: The vehicle performs time series feature fusion on the current BEV feature and the at least one historical BEV feature to generate the second BEV feature.
[0135] Since the above embodiment can generate the current BEV feature according to the currently collected environmental data, and perform time series feature fusion on the current BEV feature and the historical BEV feature to generate the second BEV feature, the driving environment perception method provided in the embodiment of the present application can make the vehicle obtain the second BEV feature according to the environmental data more accurately.
[0136] S605: The vehicle sends the location information to the server.
[0137] Correspondingly, the server receives the location information sent by the vehicle.
[0138] S606: The server extracts a first BEV feature corresponding to the location information from a BEV feature library.
[0139] S607: The server sends the first BEV feature to the vehicle.
[0140] Correspondingly, the vehicle receives the first BEV feature corresponding to the location information sent by the server.
[0141] S608: The vehicle performs feature fusion on the first BEV feature and the second BEV feature to obtain a fused BEV feature.
[0142] S609: The vehicle obtains a perception result of the driving environment based on the fused BEV features.
[0143] S610: The vehicle sends update indication information carrying the location information and the fused BEV feature to the server.
[0144] Correspondingly, the server receives update indication information sent by the vehicle, which carries the location information and the fused BEV feature.
[0145] Among them, it carries the location information and the update indication information of the fused BEV feature.
[0146] S611. In response to the update indication information, the server updates the BEV feature corresponding to the location information in the BEV feature library according to the fused BEV feature.
[0147] In some embodiments, the server updates the BEV feature corresponding to the location information in the BEV feature library according to the fused BEV feature, including: replacing the BEV feature corresponding to the location information in the BEV feature library with the fused BEV feature.
[0148] In some embodiments, the server updates the BEV feature corresponding to the location information in the BEV feature library based on the fused BEV feature, including: fusing the BEV feature corresponding to the location information in the BEV feature library and the fused BEV feature, and updating the BEV feature obtained by fusing the BEV feature corresponding to the location information in the BEV feature library and the fused BEV feature to the BEV feature corresponding to the location information.
[0149] The above-mentioned embodiment can also send the fused BEV feature to the server after the vehicle fuses the first BEV feature and the second BEV feature to obtain the fused BEV feature, so that the server updates the BEV feature corresponding to the location information in the database according to the fused BEV feature sent by the vehicle. Therefore, the above-mentioned embodiment can update the BEV feature corresponding to the location information in time to avoid changes in the driving environment corresponding to the location information, and the BEV features stored in the BEV feature library are updated accordingly.
[0150] Reference Figure 7 As shown, in some embodiments, the driving environment perception device in the vehicle that implements the driving environment perception method provided in the above embodiments may include: a feature extraction module 71, a feature fusion module 72 and a perception module 73. Among them, the feature extraction module 71 includes: a feature extraction unit 711 and a time series fusion unit 712. The feature extraction unit 711 is used to generate the corresponding current BEV feature according to the environmental data. The time series fusion unit 712 is used to spatially fuse the current BEV feature and the historical BEV feature output by the feature extraction unit 711 to obtain a second BEV feature. The feature fusion module 72 and the perception module 73 are respectively connected to Figure 3 The function of the feature fusion module 32 and the perception module 33 in the illustrated embodiment is the same and will not be described in detail here.
[0151] In some embodiments, the environmental data includes at least two sensor data acquired by at least two sensors, and generating the second BEV feature according to the environmental data includes the following steps a to d:
[0152] Step a: extracting BEV features from the at least two sensor data respectively to obtain BEV features corresponding to the at least two sensor data respectively.
[0153] Step b: performing spatial feature fusion on the BEV features corresponding to the at least two sensor data to generate a current BEV feature.
[0154] Step c: Obtain at least one historical BEV characteristic.
[0155] The historical BEV feature is a BEV feature generated according to the historical environment data corresponding to the location information.
[0156] Step d: performing time series feature fusion on the current BEV feature and the at least one historical BEV feature to generate the second BEV feature.
[0157] Reference Figure 8 As shown, in some embodiments, the driving environment perception device in the vehicle that implements the driving environment perception method provided in the above embodiment may include: a feature extraction module 81, a feature fusion module 82 and a perception module 83. Among them, the feature extraction module 81 includes: a plurality of feature extraction units 811, a spatial fusion unit 812 and a temporal fusion unit 813, the plurality of feature extraction units 811 are respectively used to generate corresponding BEV features according to a type of sensor data, the spatial fusion unit 812 is used to perform spatial fusion on the BEV features output by the plurality of feature extraction units 811 to generate the current BEV features, and the temporal fusion unit 813 is used to perform spatial fusion on the current BEV features and the historical BEV features output by the spatial fusion unit 812 to obtain the second BEV features.
[0158] In some embodiments, before the server extracts the first BEV feature corresponding to the location information from the BEV feature library, the driving environment perception method provided in the embodiment of the present application also includes: the server obtains environmental data corresponding to multiple location information based on a shadow mode; generates a BEV feature corresponding to each location information according to the environmental data corresponding to each location information; and saves the BEV feature corresponding to each location information into the BEV feature library.
[0159] Specifically, obtaining information based on the shadow mode means that when someone is driving, the vehicle system runs but does not participate in vehicle control. It only verifies the decision-making algorithm. Once the decision is inconsistent with the driver's behavior, the scene is judged as a special operating condition, which in turn triggers the upload of the collected environmental data to the server.
[0160] Reference Fig. 9 As shown, in some embodiments, the overall architecture of the driving environment perception method provided by the above embodiments includes: a vehicle 91 and a server 92. The server 92 includes: a collection database 921, a feature extraction module 922, and a BEV feature library 923. The collection database 921 is used to receive and save the location information and environmental data uploaded by the vehicle; the feature extraction module 922 is used to generate the BEV features corresponding to the location information according to the environmental data corresponding to the location information; the BEV feature library 923 is used to save the BEV features corresponding to the location information. The structure of the vehicle 91 can be similar to Figure 3 , Figure 5 , Figure 7 , Figure 8 Any of the same, Fig. 9 China-Israel Figure 8 The same is shown as an example.
[0161] Based on the same inventive concept, as an implementation of the above method, an embodiment of the present application also provides a driving assistance device, which corresponds to the above method embodiment. For ease of reading, this embodiment will no longer repeat the details of the above method embodiment one by one, but it should be clear that the driving assistance device in this embodiment can correspond to all the contents in the above method embodiment.
[0162] The present application embodiment provides a vehicle, Fig.10 is a schematic diagram of the vehicle structure, such as Fig.10 As shown, the vehicle 100 includes:
[0163] The information acquisition module 101 is used to obtain environmental data of the driving environment of the vehicle and the location information of the vehicle;
[0164] The communication module 102 is used to send the location information to a server, and receive a first bird's-eye view BEV feature corresponding to the location information sent by the server;
[0165] A feature generation module 103, configured to generate a second BEV feature according to the environmental data;
[0166] A feature fusion module 104 is used to fuse the first BEV feature and the second BEV feature to obtain a fused BEV feature;
[0167] The environment perception module 105 is used to obtain the perception result of the driving environment based on the fused BEV features.
[0168] As an optional implementation of the embodiment of the present application, the communication module 102 is also used to send update indication information carrying the location information and the fused BEV feature to the server after generating the second BEV feature according to the environmental data; wherein, the update indication information carrying the location information and the fused BEV feature.
[0169] As an optional implementation of the embodiment of the present application, the environmental data includes at least two sensor data obtained by at least two sensors; the feature generation module 103 is specifically used to extract BEV features from the at least two sensor data respectively, obtain BEV features corresponding to the at least two sensor data respectively, and perform spatial feature fusion on the BEV features corresponding to the at least two sensor data respectively to generate the second BEV feature.
[0170] As an optional implementation of the embodiment of the present application, the feature generation module 103 is specifically used to generate a current BEV feature based on the environmental data; obtain at least one historical BEV feature; the historical BEV feature is to generate a BEV feature based on the historical environmental data corresponding to the location information, and perform time series feature fusion on the current BEV feature and the at least one historical BEV feature to generate the second BEV feature.
[0171] As an optional implementation of the embodiment of the present application,
[0172] The environmental data includes: at least one of image data, laser radar data, millimeter wave radar data, and distance sensor data;
[0173] The perception result includes: at least one of road structural elements, traffic lights, and road topological relationships.
[0174] The embodiment of the present application provides a server, Fig.11 is a schematic diagram of the server structure, such as Fig.11 As shown, the server 200 includes:
[0175] Receiving module 201, used for receiving location information sent by the vehicle;
[0176] A processing module 202 is used to extract a first bird's-eye view BEV feature corresponding to the position information from a BEV feature library;
[0177] The sending module 203 is used to send the first BEV feature to the vehicle, so that the vehicle fuses the first BEV feature with a second BEV feature generated according to environmental data of the driving environment of the vehicle to obtain a fused BEV feature, and obtains a perception result of the driving environment based on the fused BEV feature.
[0178] As an optional implementation of the embodiment of the present application,
[0179] The receiving module 201 is further used to receive update indication information sent by the vehicle and carrying the location information and the fused BEV feature;
[0180] The processing module 202 is further configured to update the BEV feature corresponding to the location information in the BEV feature library according to the fused BEV feature in response to the update indication information.
[0181] As an optional implementation of the embodiment of the present application, the processing unit 203 is also used to obtain environmental data corresponding to multiple location information based on the shadow mode before extracting the first BEV feature corresponding to the location information from the BEV feature library; generate BEV features corresponding to each location information according to the environmental data corresponding to each location information; and save the BEV features corresponding to each location information into the BEV feature library.
[0182] Based on the same inventive concept, an embodiment of the present application also provides an electronic device. Fig.12 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application, such as Fig.12 As shown, the electronic device provided in this embodiment includes: a memory 121 and a processor 122, wherein the memory 121 is used to store a computer program; and the processor 122 is used to execute the driving environment perception method provided in the above embodiment when executing the computer program.
[0183] Based on the same inventive concept, an embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the computing device implements the driving environment perception method provided in the above embodiment.
[0184] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media that include computer-usable program code.
[0185] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.
[0186] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0187] Computer readable media include permanent and non-permanent, removable and non-removable storage media. Storage media can be implemented by any method or technology to store information, and the information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0188] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.
Claims
1. A driving environment perception method, characterized in that: include: Acquiring environmental data of a vehicle's driving environment and location information of the vehicle; Sending the location information to a server; Receiving a first bird's-eye view (BEV) feature corresponding to the location information sent by the server; generating a second BEV characteristic based on the environmental data; Performing feature fusion on the first BEV feature and the second BEV feature to obtain a fused BEV feature; A perception result of the driving environment is obtained based on the fused BEV feature.
2. The method according to claim 1, characterized in that After generating the second BEV characteristic according to the environmental data, the method further includes: Sending update indication information carrying the location information and the fused BEV feature to the server; The update indication information is used to instruct the server to update the BEV feature corresponding to the location information according to the fused BEV feature.
3. The method according to claim 1, characterized in that The environmental data includes at least two types of sensor data acquired by at least two types of sensors; and generating a second BEV feature according to the environmental data includes: Extracting BEV features from the at least two sensor data respectively to obtain BEV features corresponding to the at least two sensor data respectively; Perform spatial feature fusion on the BEV features respectively corresponding to the at least two sensor data to generate the second BEV feature.
4. The method according to claim 1, characterized in that: The generating a second BEV feature according to the environmental data comprises: generating a current BEV characteristic based on the environmental data; Acquire at least one historical BEV feature; the historical BEV feature is a BEV feature generated according to the historical environmental data corresponding to the location information; Performing time series feature fusion on the current BEV feature and the at least one historical BEV feature to generate the second BEV feature.
5. The method according to any one of claims 1 to 4, characterized in that: The environmental data includes: at least one of image data, laser radar data, millimeter wave radar data, and distance sensor data; The perception result includes: at least one of road structural elements, traffic lights, and road topological relationships.
6. A driving environment perception method, characterized in that: include: Receive location information sent by the vehicle; Extracting a first bird's-eye view BEV feature corresponding to the location information from a BEV feature library; The first BEV feature is sent to the vehicle so that the vehicle fuses the first BEV feature with a second BEV feature generated according to environmental data of the driving environment of the vehicle to obtain a fused BEV feature, and obtains a perception result of the driving environment based on the fused BEV feature.
7. The method according to claim 6, characterized in that The method further comprises: receiving update indication information sent by the vehicle and carrying the location information and the fused BEV feature; In response to the update indication information, the BEV feature corresponding to the location information in the BEV feature library is updated according to the fused BEV feature.
8. The method according to claim 6 or 7, characterized in that: Before extracting the first BEV feature corresponding to the location information from the BEV feature library, the method further includes: Environmental data corresponding to multiple location information obtained based on shadow mode; Generate BEV features corresponding to each position information according to the environmental data corresponding to each position information; The BEV features corresponding to each piece of position information are saved in the BEV feature library.
9. A vehicle, characterized in that: include: An information acquisition module, used to obtain environmental data of the vehicle's driving environment and location information of the vehicle; a communication module, configured to send the location information to a server, and receive a first bird's-eye view BEV feature corresponding to the location information sent by the server; a feature generation module, configured to generate a second BEV feature according to the environmental data; A feature fusion module, used for fusing the first BEV feature and the second BEV feature to obtain a fused BEV feature; An environment perception module is used to obtain a perception result of the driving environment based on the fused BEV features.
10. A server, characterized in that: include: A receiving module, used for receiving location information sent by the vehicle; A processing module, configured to extract a first bird's-eye view BEV feature corresponding to the position information from a BEV feature library; A sending module is used to send the first BEV feature to the vehicle, so that the vehicle fuses the first BEV feature with a second BEV feature generated according to environmental data of the driving environment of the vehicle to obtain a fused BEV feature, and obtain a perception result of the driving environment based on the fused BEV feature.
11. An electronic device, characterized in that: include: A memory and a processor, wherein the memory is used to store a computer program; and the processor is used to enable the electronic device to implement the driving environment perception method described in any one of claims 1 to 8 when executing the computer program.
12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a computing device, the computing device implements the driving environment perception method according to any one of claims 1 to 8.