Information fusion method, device, equipment and storage medium

Abstract

The present disclosure provides an information fusion method and device, equipment and a storage medium, relating to the technical field of artificial intelligence, in particular to the technical field of automatic driving. The specific implementation scheme is: obtaining a system environment variable of an automatic driving vehicle, determining a target fusion algorithm corresponding to a fusion computing platform according to the system environment variable of the automatic driving vehicle, when receiving multi-type sensor data of the automatic driving vehicle, performing information fusion on the multi-type sensor data according to the target fusion algorithm to obtain target sensor data after information fusion. In this way, the target fusion algorithm suitable for the automatic driving vehicle can be determined, and then the multi-type sensor data is fused according to the target fusion algorithm, which can effectively improve the accuracy of information fusion.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and more particularly to the field of autonomous driving technology, specifically to an information fusion method, apparatus, device, and storage medium. Background Technology

[0002] Autonomous vehicles are intelligent vehicles that perceive the road environment through onboard sensor systems, automatically plan their routes, and control the vehicle to reach predetermined destinations. These onboard sensor systems can include various types of sensors, such as video cameras and radar sensors. Typically, data from these different types of sensors needs to be fused to allow for subsequent driving planning. Summary of the Invention

[0003] This disclosure provides an information fusion method, apparatus, device, and storage medium.

[0004] According to one aspect of this disclosure, an information fusion method is provided, the method comprising:

[0005] The system environment variables of the autonomous vehicle are obtained. These system environment variables are used to indicate the system environment of the fusion computing platform on which the autonomous vehicle is running. Different fusion computing platforms correspond to different sensor combinations, and these sensor combinations correspond to multiple types of sensors.

[0006] Based on the system environment variables of the autonomous vehicle, a target fusion algorithm corresponding to the fusion computing platform is determined. This target fusion algorithm is used to perform information fusion based on the sensor combination corresponding to the fusion computing platform. Different fusion computing platforms correspond to different fusion algorithms.

[0007] Upon receiving multi-type sensor data from the autonomous vehicle, the target fusion algorithm is used to fuse the multi-type sensor data to obtain the fused target sensor data, which is then used for autonomous driving operations.

[0008] According to another aspect of this disclosure, an information fusion apparatus is provided, the apparatus comprising:

[0009] The acquisition module is used to acquire the system environment variables of the autonomous vehicle. These system environment variables indicate the system environment of the fusion computing platform on which the autonomous vehicle is running. Different fusion computing platforms correspond to different sensor combinations, and these sensor combinations correspond to multiple types of sensors.

[0010] The determination module is used to determine the target fusion algorithm corresponding to the fusion computing platform based on the system environmental variables of the autonomous vehicle. The target fusion algorithm is used to perform information fusion based on the sensor combination corresponding to the fusion computing platform; different fusion computing platforms correspond to different fusion algorithms.

[0011] The fusion module is used to fuse the multi-type sensor data received from the autonomous vehicle according to the target fusion algorithm to obtain the fused target sensor data, which is used for autonomous driving operations.

[0012] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0013] At least one processor; and

[0014] The memory is communicatively connected to the at least one processor; wherein,

[0015] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the information fusion method provided in this disclosure.

[0016] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform the information fusion method provided in this disclosure.

[0017] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the information fusion method provided in this disclosure.

[0018] The technical solution provided in this disclosure can determine the target fusion algorithm corresponding to the fusion computing platform running on the autonomous vehicle by obtaining the system environmental variables of the autonomous vehicle. Since different fusion computing platforms correspond to different sensor combinations and different fusion algorithms, the target fusion algorithm determined based on the system environmental variables of the autonomous vehicle is a fusion algorithm that is compatible with the current autonomous vehicle. That is, it is a fusion algorithm that can be applied to multiple types of sensors in the current autonomous vehicle. Therefore, when receiving multiple types of sensor data from the autonomous vehicle, information fusion is performed on the multiple types of sensor data according to the target fusion algorithm, which can effectively improve the accuracy of information fusion.

[0019] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0020] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0021] Figure 1 This is a schematic diagram illustrating the implementation environment of an information fusion method according to an embodiment of this disclosure;

[0022] Figure 2 This is a schematic flowchart illustrating an information fusion method according to an embodiment of this disclosure;

[0023] Figure 3 This is a schematic flowchart illustrating an information fusion method according to an embodiment of this disclosure;

[0024] Figure 4 This is a schematic diagram illustrating the framework of an information fusion method according to an embodiment of this disclosure;

[0025] Figure 5 This is a schematic diagram illustrating a dynamic obstacle fusion process according to an embodiment of this disclosure;

[0026] Figure 6 This is a structural block diagram of an information fusion device shown in an embodiment of this disclosure;

[0027] Figure 7 This is a block diagram of an electronic device used to implement the information fusion method of the embodiments of this disclosure. Detailed Implementation

[0028] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0029] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0030] First, the application scenarios involved in the embodiments of this disclosure are described. The information fusion method provided in the embodiments of this disclosure can be applied to autonomous driving scenarios. For example, the information fusion method provided in the embodiments of this disclosure can be applied to autonomous valet parking (AVP) scenarios or other autonomous driving scenarios.

[0031] The information fusion method provided in this disclosure is a multi-sensor information fusion (MSIF) method. MSIF refers to fusing sensor data collected by various sensors in an autonomous vehicle, such as video cameras, lidar sensors, millimeter-wave radar sensors, and ultrasonic radar sensors. Computer technology is then used to analyze and synthesize the sensor data from these multiple sensors to complete the information processing required for driving planning, thereby more accurately and reliably describing the road environment and improving the accuracy of driving planning.

[0032] It's important to note that in related technologies, Tier One suppliers of autonomous vehicles typically need to select the appropriate fusion computing platform and sensor combination based on the vehicle model during the configuration phase. However, since Tier One suppliers often receive orders from multiple automakers for different models, and different models use different fusion computing platforms and sensor combinations, selecting the appropriate fusion computing platform and sensor combination for each model is time-consuming and inefficient.

[0033] Based on this, this disclosure provides an information fusion method. By acquiring the system environment variables of an autonomous vehicle, the target fusion algorithm corresponding to the fusion computing platform running the autonomous vehicle can be determined. Since different fusion computing platforms correspond to different sensor combinations and different fusion algorithms, the target fusion algorithm determined based on the system environment variables of the autonomous vehicle is a fusion algorithm that is compatible with the current autonomous vehicle, that is, a fusion algorithm applicable to multiple types of sensors in the current autonomous vehicle. Therefore, when receiving multiple types of sensor data from the autonomous vehicle, information fusion is performed on the multiple types of sensor data according to the target fusion algorithm, which can effectively improve the accuracy of information fusion. Compared with related technologies that manually select fusion computing platforms and sensor combinations corresponding to different vehicle models, by designing a unified autonomous driving information fusion technology that is compatible with different fusion computing platforms and different sensor combinations, not only can the difficulty of personnel development and maintenance be reduced, but the cost of the overall autonomous driving solution can also be reduced, thereby enabling support for multiple different autonomous driving products with minimal personnel costs.

[0034] Figure 1 This is a schematic diagram illustrating the implementation environment of an information fusion method according to an embodiment of this disclosure. See also... Figure 1 The implementation environment includes electronic device 101 and server 102.

[0035] The electronic device 101 is at least one of the following: in-vehicle device, smartphone, smartwatch, desktop computer, laptop, virtual reality terminal, augmented reality terminal, wireless terminal, and laptop computer. In some embodiments, the electronic device 101 has communication capabilities and can access a wired or wireless network.

[0036] In this embodiment of the disclosure, the electronic device 101 is used to acquire the system environment variables of the autonomous vehicle, determine the target fusion algorithm corresponding to the fusion computing platform based on the system environment variables of the autonomous vehicle, and when receiving multi-type sensor data of the autonomous vehicle, perform information fusion on the multi-type sensor data according to the target fusion algorithm to obtain the information-fused target sensor data.

[0037] In some embodiments, server 102 is an independent physical server, or a server cluster or distributed file system composed of multiple physical servers, or at least one of cloud servers that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and big data or artificial intelligence platforms. This disclosure does not limit the specific implementation of the embodiments.

[0038] In this embodiment of the disclosure, server 102 is used to provide communication services to the aforementioned electronic device 101. In some embodiments, the number of servers 102 may be more or fewer, and this embodiment of the disclosure is not limited in this regard. Of course, server 102 may also include other functions to provide more comprehensive and diversified services.

[0039] The following is based on Figure 1 The implementation environment shown will be used to describe the methods provided in the embodiments of this disclosure.

[0040] Figure 2 This is a schematic flowchart illustrating an information fusion method according to an embodiment of this disclosure. In some embodiments, the information fusion method is performed by an electronic device. Figure 2 As shown, the method includes the following steps.

[0041] S201. Obtain the system environment variables of the autonomous vehicle. These system environment variables are used to indicate the system environment of the fusion computing platform on which the autonomous vehicle is running. Different fusion computing platforms correspond to different sensor combinations, and these sensor combinations correspond to multiple types of sensors.

[0042] The sensor data described herein is data used to describe obstacle targets around the autonomous vehicle. In some embodiments, the sensors installed in the autonomous vehicle may include at least two of the following: a fisheye camera, a wide-angle camera, a LiDAR sensor, a millimeter-wave radar sensor, and an ultrasonic radar sensor. Accordingly, the various types of sensor data from the autonomous vehicle may include at least two of the following: fisheye camera data, wide-angle camera data, LiDAR sensor data, millimeter-wave radar sensor data, and ultrasonic radar sensor data.

[0043] System environment variables are used to indicate the system environment of the current autonomous vehicle; specifically, they refer to parameters used in the platform system to specify the system's operating environment. For example, system environment variables may be parameters used to specify file locations, file paths, processor descriptions, or the system root directory. In some embodiments, system environment variables may be in the form of variables or strings.

[0044] S202. Based on the system environment variables of the autonomous vehicle, determine the target fusion algorithm corresponding to the fusion computing platform. The target fusion algorithm is used to perform information fusion based on the sensor combination corresponding to the fusion computing platform; different fusion computing platforms correspond to different fusion algorithms.

[0045] In this embodiment of the disclosure, the target fusion algorithm is used to refer to a fusion algorithm adapted to the current autonomous vehicle. Thus, the target fusion algorithm determined based on the system environment variables of the autonomous vehicle is a fusion algorithm that fits the current autonomous vehicle, that is, a fusion algorithm that can be applied to multiple types of sensors in the current autonomous vehicle.

[0046] S203. Upon receiving multi-type sensor data from the autonomous vehicle, the multi-type sensor data is fused according to the target fusion algorithm to obtain the fused target sensor data, which is used for autonomous driving operations.

[0047] The technical solution provided in this disclosure can determine the target fusion algorithm corresponding to the fusion computing platform running the autonomous vehicle by obtaining the system environmental variables of the autonomous vehicle. Since different fusion computing platforms correspond to different sensor combinations and different fusion algorithms, the target fusion algorithm determined according to the system environmental variables of the autonomous vehicle is a fusion algorithm that is compatible with the current autonomous vehicle. That is, it is a fusion algorithm that can be applied to multiple types of sensors in the current autonomous vehicle. Therefore, when receiving multiple types of sensor data from the autonomous vehicle, information fusion is performed on the multiple types of sensor data according to the target fusion algorithm, which can effectively improve the accuracy of information fusion.

[0048] The above Figure 2As a simple embodiment shown in this disclosure, the information fusion method provided in this disclosure will be described below based on a specific embodiment. Figure 3 This is a schematic flowchart illustrating an information fusion method according to an embodiment of this disclosure. In some embodiments, the information fusion method is performed by an electronic device. For example, the electronic device can be the one described above. Figure 1 The vehicle-mounted equipment shown. (For example...) Figure 3 As shown, with an electronic device as the executing entity, the method includes the following steps.

[0049] S301. In response to receiving any type of sensor data from multiple types of sensor data from an autonomous vehicle, the electronic device performs data preprocessing on the data to obtain preprocessed multiple types of sensor data.

[0050] The sensor data described herein is used to describe obstacle targets around the autonomous vehicle. In some embodiments, the sensors installed in the autonomous vehicle may include at least two of the following: fisheye camera, wide-angle camera, LiDAR sensor, millimeter-wave radar sensor, and ultrasonic radar sensor. Of course, in other embodiments, the autonomous vehicle may also be equipped with other types of sensors, and the present disclosure does not limit the sensor configuration.

[0051] Accordingly, the various sensor data of the aforementioned autonomous vehicle may include at least two of the following: fisheye camera data, wide-angle camera data, LiDAR sensor data, millimeter-wave radar sensor data, and ultrasonic radar sensor data. Specifically, fisheye camera data may include a feasible free space point cloud and two-dimensional (2D) detection boxes for obstacles. Wide-angle camera data may include three-dimensional (3D) detection boxes for obstacles. LiDAR sensor data may include LiDAR point clouds and three-dimensional (3D) detection boxes for obstacles. Millimeter-wave radar sensor data may include millimeter-wave radar point clouds and trajectory data of obstacles. Ultrasonic radar sensor data may include ultrasonic point clouds.

[0052] In some embodiments, if an electronic device detects a processing message for any type of sensor data from an autonomous vehicle, it preprocesses that type of sensor data to obtain preprocessed data. Thus, upon detecting a processing message for any type of sensor data, the preprocessing process for that type of sensor data is triggered, enabling timely preprocessing of sensor data. Furthermore, by preprocessing multiple types of sensor data, information with unified data types or rich data content can be obtained, facilitating subsequent information fusion using the preprocessed multi-type sensor data and improving the accuracy of information fusion.

[0053] In some embodiments, the data preprocessing process described above may include at least one of (1-1) to (1-5) below. The data preprocessing process will be illustrated below with examples of fisheye camera data, wide-angle camera data, LiDAR sensor data, millimeter-wave radar sensor data, and ultrasonic radar sensor data:

[0054] (1-1) The electronic device performs outlier filtering on the point cloud data of obstacle targets in the data of any type of sensor.

[0055] The obstacle target can be a vehicle, bicycle, pedestrian, or other roadblock, etc. Point cloud data refers to a dataset of points in a specific coordinate system. Accordingly, in this embodiment of the disclosure, the point cloud data of the obstacle target is also a dataset of points of the obstacle target in the road coordinate system. In some embodiments, the point cloud data is used to characterize the position of the obstacle target.

[0056] Outlier filtering refers to filtering out outlier samples in data. In some embodiments, electronic devices may employ statistical methods, clustering methods, or one-class support vector machines (SVMs) to perform the outlier filtering process. This disclosure does not limit the method used for outlier filtering.

[0057] In some embodiments, taking the point cloud data of the obstacle target as an example of feasible region point cloud from fisheye camera data, the above (1-1) can be replaced by: the electronic device performing outlier filtering on the feasible region point cloud from the fisheye camera data. Alternatively, in other embodiments, taking the point cloud data of the obstacle target as an example of ultrasonic point cloud from ultrasonic radar sensor data, the above (1-1) can be replaced by: the electronic device performing outlier filtering on the ultrasonic point cloud from the ultrasonic radar sensor data. Of course, depending on the type of sensors used in the autonomous vehicle, the point cloud data of the obstacle target can also be point cloud data from other sensor data. This disclosure does not limit the type of point cloud data.

[0058] In the above embodiments, by filtering outliers, abnormal data can be filtered quickly and accurately, ensuring the accuracy of point cloud data and avoiding the impact of abnormal data on subsequent information fusion, thereby improving the accuracy of information fusion.

[0059] (1-2) The electronic device fits the bounding box of the obstacle target based on the point cloud data of the obstacle target in the data of any type of sensor to obtain the fitted bounding box of the obstacle target.

[0060] In this context, the bounding box fitted to the obstacle target is used to represent the obstacle target. Bounding box fitting refers to transforming scattered point cloud data into regular objects using bounding boxes, so that the constructed bounding boxes can be used to extract the geometric attributes of the obstacle target.

[0061] In some embodiments, taking the point cloud data of the obstacle target as the feasible domain point cloud in the fisheye camera data as an example, the above (1-2) can be replaced by: the electronic device performs bounding box fitting on the feasible domain point cloud in the fisheye camera data to obtain the fitted bounding box of the obstacle target.

[0062] In the above embodiments, by fitting bounding boxes, obstacle targets with regular object shapes are obtained, so that the fitted bounding boxes of the obstacle targets can be used for information fusion in the future, thereby improving the accuracy and comprehensiveness of information fusion.

[0063] (1-3) The electronic device determines the orientation of the obstacle target based on the two-dimensional detection box of the obstacle target in the data of any type of sensor.

[0064] In some embodiments, the electronic device extracts grounding point information from the two-dimensional detection frame of the obstacle target from the multi-sensor data, and determines the orientation of the obstacle target based on the grounding point information. Here, grounding point information refers to the grounding point information of the obstacle target. For example, taking a vehicle as an example, the grounding point information could be the tire grounding point information of the vehicle.

[0065] In some embodiments, taking the two-dimensional detection box of the obstacle target as an example, which is a two-dimensional detection box in the fisheye camera data, the above (1-3) can be replaced by: the electronic device determining the orientation of the obstacle target based on the two-dimensional detection box of the obstacle target in the fisheye camera data. Of course, depending on the type of sensors installed in the autonomous vehicle, the two-dimensional detection box of the obstacle target can also be a two-dimensional detection box in other sensor data. This disclosure does not limit the type of two-dimensional detection box.

[0066] In the above embodiments, the orientation of the obstacle target is determined based on the two-dimensional detection box of the obstacle target, so that the orientation of the obstacle target can be used for subsequent information fusion (i.e. orientation fusion), thereby improving the accuracy and comprehensiveness of information fusion.

[0067] (1-4) The electronic device performs Kalman filtering to track the three-dimensional detection box of the obstacle target in the data of any type of sensor to obtain the velocity of the obstacle target.

[0068] Kalman filtering tracking is an optimal estimation method that processes data containing observation noise and interference signals to obtain estimated values ​​for various parameters (such as speed). It should be understood that autonomous vehicles are subject to various uncertainties during operation, which can cause sensor data to contain random noise. Therefore, to ensure the accuracy of sensor data, Kalman filtering is necessary to process it.

[0069] In some embodiments, taking the 3D detection bounding box of the obstacle target as an example of a 3D detection bounding box in wide-angle camera data, the above (1-4) can be replaced by: the electronic device performing Kalman filtering tracking on the 3D detection bounding box in the wide-angle camera data to obtain the velocity of the obstacle target. Alternatively, in other embodiments, taking the 3D detection bounding box of the obstacle target as an example of a 3D detection bounding box in lidar sensor data, the above (1-4) can be replaced by: the electronic device performing Kalman filtering tracking on the 3D detection bounding box in the lidar sensor data to obtain the velocity of the obstacle target. Of course, depending on the type of sensors used in the autonomous vehicle, the 3D detection bounding box of the obstacle target can also be a 3D detection bounding box from other sensor data. This disclosure does not limit the type of 3D detection bounding box.

[0070] In the above embodiments, Kalman filtering tracking can effectively filter out noise carried in sensor data, effectively ensuring the accuracy of sensor data, thereby improving the accuracy of information fusion and also improving the comprehensiveness of information fusion.

[0071] (1-5) The electronic device determines the position, category and size of the obstacle target based on the point cloud data and track data of the obstacle target in the data of any type of sensor.

[0072] In some embodiments, the point cloud data of the obstacle target in (1-5) above can be millimeter-wave radar point cloud data from millimeter-wave radar sensor data, and the trajectory data of the obstacle target can be trajectory data from millimeter-wave radar sensor data. The attribute information of each point in the millimeter-wave radar point cloud can include the speed of the obstacle target and the distance between the obstacle target and the current autonomous vehicle. The attribute information of each trajectory point in the trajectory data includes the obstacle target's position, speed, angle, radar cross-section (RCS), etc.

[0073] In some embodiments, the electronic device employs a convolutional neural network to determine the location, category, and size of the obstacle target. The corresponding process includes: the electronic device inputting point cloud data and trajectory data of the obstacle target from multi-class sensor data into the convolutional neural network; and processing the point cloud data and trajectory data through the convolutional neural network to obtain the location, category, and size of the obstacle target. The location of the obstacle target can be in the form of a detection box (two-dimensional or three-dimensional). The category of the obstacle target can include a category and a category probability, where the category probability represents the likelihood that the obstacle target belongs to the corresponding category.

[0074] In the above embodiments, by determining the location, category, and size of the obstacle target, the location, category, and size of the obstacle target can be used in the subsequent information fusion process, thereby improving both the accuracy and comprehensiveness of information fusion.

[0075] In some embodiments, the electronic device is provided with a data preprocessing module, which is used to preprocess any type of sensor data among multiple types of sensor data received from an autonomous vehicle to obtain preprocessed multiple types of sensor data.

[0076] Accordingly, in some embodiments, the aforementioned electronic device includes at least two of the following: a fisheye camera data preprocessing module, a wide-angle camera data preprocessing module, a lidar data preprocessing module, a millimeter-wave radar data preprocessing module, and an ultrasonic radar data preprocessing module. Specifically, the fisheye camera data preprocessing module is used to perform outlier filtering and bounding box fitting on the feasible region point cloud in the fisheye camera data, and to determine the orientation of the obstacle target based on the two-dimensional detection box of the obstacle target. The wide-angle camera data preprocessing module is used to perform Kalman filter tracking on the three-dimensional detection box in the wide-angle camera data. The lidar data preprocessing module is used to perform Kalman filter tracking on the three-dimensional detection box in the lidar sensor data. The millimeter-wave radar data preprocessing module is used to determine the position, category, and size of the obstacle target. The ultrasonic radar data preprocessing module is used to perform outlier filtering on the ultrasonic point cloud in the ultrasonic radar sensor data.

[0077] It should be noted that when performing data preprocessing on the multiple types of sensor data based on S301, the corresponding sensor preprocessing module needs to be triggered to execute the corresponding data preprocessing process according to the type of sensor data received. Accordingly, the above S301 can be replaced by: the electronic device, in response to receiving any type of sensor data from the multiple types of sensor data of the autonomous vehicle, triggering the sensor preprocessing module corresponding to that type of sensor data to perform data preprocessing on that type of sensor data, obtaining the preprocessed data of that type of sensor data. It should be understood that if the autonomous vehicle is equipped with a fisheye camera, a wide-angle camera, and a LiDAR sensor, then only the fisheye camera data preprocessing module, the wide-angle camera data preprocessing module, and the LiDAR data preprocessing module need to be triggered to execute the corresponding data preprocessing process.

[0078] In some embodiments, before implementing this solution, technicians will add a data message trigger configuration file to the data preprocessing module of the electronic device. This data message trigger configuration file indicates that the triggering mechanism of the data preprocessing module is message-triggered; that is, when a processing message for sensor data is received, the executor code of the data preprocessing module is triggered. For example, when the electronic device receives a processing message for fisheye camera data, the fisheye camera data preprocessing module is triggered to execute the corresponding data preprocessing procedure. Alternatively, when the electronic device receives a processing message for wide-angle camera data, the wide-angle camera data preprocessing module is triggered to execute the corresponding data preprocessing procedure.

[0079] S302. The electronic device obtains the system environment variables of the autonomous vehicle, which are used to indicate the system environment of the fusion computing platform on which the autonomous vehicle is running.

[0080] System environment variables are used to indicate the system environment of the current autonomous vehicle, specifically referring to parameters used in the platform system to specify the system operating environment. For example, system environment variables may be parameters used to specify file locations, file paths, processor descriptions, or system root directories. In some embodiments, system environment variables may be in the form of variables or strings.

[0081] In this embodiment of the disclosure, different fusion computing platforms correspond to different sensor combinations, which may include multiple types of sensors. For example, taking a simple configuration fusion computing platform as an example, its corresponding sensor combination may be a combination of a fisheye camera and a wide-angle camera. Taking a high-configuration fusion computing platform as an example, its corresponding sensor combination may be a combination of a fisheye camera, a wide-angle camera, and a radar sensor.

[0082] S303. The electronic device determines the target fusion algorithm corresponding to the fusion computing platform based on the system environmental variables of the autonomous vehicle. The target fusion algorithm is used to perform information fusion based on the sensor combination corresponding to the fusion computing platform.

[0083] In this embodiment of the disclosure, different fusion computing platforms correspond to different fusion algorithms. Thus, the target fusion algorithm determined based on the system environment variables of the autonomous vehicle is a fusion algorithm that is compatible with the current autonomous vehicle, that is, a fusion algorithm that can be applied to multiple types of sensors in the current autonomous vehicle.

[0084] In some embodiments, the electronic device determines the fusion computing platform running on the autonomous vehicle based on the system environment variables of the autonomous vehicle, and obtains the algorithm configuration information corresponding to the identification information from a preset correspondence based on the identification information of the fusion computing platform running on the autonomous vehicle, as the target fusion algorithm. The preset correspondence includes multiple identification information and the algorithm configuration information corresponding to the multiple identification information.

[0085] The identification information can be an identification field of the converged computing power platform, which is a compilation field used to uniquely identify the converged computing power platform. For example, this identification field can be a string. It should be understood that one converged computing power platform corresponds to one identification field. Thus, by setting different compilation fields for different converged computing power platforms at the system level, it is possible to identify different converged computing power platforms at the functional code level, so that the identification field can be used to compile and read algorithm configuration information later. Algorithm configuration information can be configuration parameters, configuration fields, configuration files, or the entire configuration file.

[0086] In the above embodiments, by maintaining multiple identification information and the algorithm configuration information corresponding to the multiple identification information in a preset correspondence, after determining the identification information of the fusion computing platform, the corresponding algorithm configuration information can be read in the preset correspondence based on the identification information, thereby quickly obtaining the target fusion algorithm that matches the fusion computing platform of the autonomous vehicle and improving the accuracy of information fusion.

[0087] In some embodiments, the process of an electronic device determining a fusion computing platform may involve: determining a state flag corresponding to the system environment variables of the autonomous vehicle, and determining the fusion computing platform operated by the autonomous vehicle based on the state flag corresponding to the system environment variables. The state flag indicates the type of the fusion computing platform. In some embodiments, the state flag can be in the form of a variable or a string. Thus, by setting the state flag, the type of the fusion computing platform can be quickly determined, enabling the acquisition of the corresponding target fusion algorithm using the determined fusion computing platform, thereby ensuring the efficiency of information fusion.

[0088] In some embodiments, when the fusion computing platform of an autonomous vehicle changes, the electronic device can also switch the fusion algorithm based on a state flag. The corresponding process is as follows: when the system's environmental variables change, the electronic device, in response to the state flag switching from the original flag to a target flag, switches the target fusion algorithm to the fusion algorithm corresponding to the target flag. Thus, by setting different state flags for different fusion computing platforms, switching between different fusion algorithms can be achieved, thereby ensuring compatibility of the fusion algorithm with different fusion computing platforms and different sensor combinations.

[0089] S304. When receiving multi-type sensor data from the autonomous vehicle, the electronic device performs information fusion on the pre-processed multi-type sensor data according to the target fusion algorithm to obtain the fused target sensor data, which is used for autonomous driving operations.

[0090] In some embodiments, the above information fusion process may be: the electronic device determines the target compilation option of the target fusion algorithm, and performs information fusion on the multi-type sensor data according to the fusion sub-algorithm corresponding to the target compilation option.

[0091] The target compilation option matches the sensor combination corresponding to the fusion computing platform, and one of the compilation options in the target compilation option corresponds to the fusion sub-algorithm of a type of sensor in the sensor combination.

[0092] In this embodiment of the disclosure, different fusion computing platforms correspond to different compilation options. In some embodiments, a compilation option can be a macro definition. Thus, by setting different compilation options for different fusion computing platforms, the fusion algorithms of different fusion computing platforms can be distinguished at the functional level. The macro definition determines which code to compile, thereby obtaining the output (such as an executable file) of the corresponding fusion computing platform.

[0093] The process of information fusion will be described below based on at least one of (2-1) to (2-4) below:

[0094] (2-1) The electronic device determines the occupied grid map based on the point cloud data of the obstacle targets in the multi-type sensor data. The occupied grid map is used to present the distribution of multiple obstacle targets in the form of a grid map.

[0095] In some embodiments, the electronic device constructs an initial grid map in the area surrounding the current autonomous vehicle; based on the position of each point in the point cloud data of the obstacle target, it determines whether each point in the point cloud data is located within a grid of the initial grid map; if a point is located in any grid, the occupancy probability value of that grid is increased by a preset value; the occupancy probability value of each grid in the initial grid map is counted, and grids whose occupancy probability value reaches a preset threshold are determined as occupied grids, and grids whose occupancy probability value does not reach the preset threshold are determined as unoccupied grids; based on the grids whose occupancy probability value reaches the preset threshold, an occupied grid map is drawn.

[0096] In this context, the initial grid map refers to a mesh map. In some embodiments, an initial grid map is drawn within an area centered on the autonomous vehicle and at a preset distance from the vehicle, according to a preset grid spacing. The preset distance is a fixed distance set in advance, such as 5 meters or 10 meters. The preset spacing is a preset spacing size, such as 1 meter or 2 meters. This disclosure does not limit the settings of the preset distance and preset size.

[0097] The occupancy probability value is used to characterize the probability that a corresponding grid has been occupied. In some embodiments, if a point exists in any grid, the occupancy probability value of that grid is increased by a preset value. The preset value is a fixed value set in advance, such as 0.1. This disclosure does not limit the setting of the preset value.

[0098] The preset threshold is a pre-defined fixed threshold, such as 0.8. For example, if the occupancy probability of a grid is 0.8 or higher, the grid is considered occupied; if the occupancy probability is lower than 0.8, the grid is considered unoccupied. Further, in some embodiments, for grids already occupied in the initial raster map, a point can be marked on that grid, and then after traversing all points in the point cloud data, an occupied raster map can be drawn.

[0099] In the above embodiments, by drawing an occupation grid map, the distribution of multiple obstacle targets can be presented in the form of a grid map, which can effectively reflect the distribution of obstacle targets and improve the effect of information fusion.

[0100] (2-2) The electronic device performs information fusion on the multi-sensor data based on the three-dimensional motion trajectory of the obstacle target in the multi-sensor data. The three-dimensional motion trajectory is used to present the motion trajectory of the obstacle target in the three-dimensional image.

[0101] In some embodiments, the electronic device tracks and matches the obstacle target based on the observation boxes in the three-dimensional motion trajectory of the obstacle target in the multi-type sensor data to determine the observation boxes that match the obstacle target. Based on the observation boxes that match the obstacle target, the motion information and orientation information of the obstacle target are updated using a Kalman filter.

[0102] The motion information includes position, velocity, and acceleration. Orientation information includes direction angle. In some embodiments, a Kalman filter is used to perform the process of updating the motion and orientation information of the obstacle target.

[0103] In some embodiments, the tracking matching includes at least one of the following: cascade matching of fisheye camera data, detection box similarity matching of wide-angle camera data, Mahalanobis distance matching of millimeter-wave radar sensor data, detection box similarity matching of lidar sensor data, and geometric similarity matching of ultrasonic radar sensor data.

[0104] In some embodiments, taking the detection box (two-dimensional detection box, three-dimensional detection box, or fitted bounding box) of an obstacle target in wide-angle camera data or lidar sensor data as an example, the above-mentioned tracking and matching process may include: determining the similarity between the detection box and the observation box of the obstacle target based on the observation box in the three-dimensional motion trajectory of the obstacle target; associating and matching the detection box and the observation box with similarity reaching a similarity threshold; and then updating the motion information and orientation information of the detection box using the motion information and orientation information of the observation box.

[0105] In other embodiments, taking the track points of an obstacle target in millimeter-wave radar sensor data as an example, the above-mentioned tracking and matching process may include: determining the Mahalanobis distance between the track points of the obstacle target and the observation frame based on the observation frame in the three-dimensional motion trajectory of the obstacle target; associating and matching track points with Mahalanobis distance less than a preset distance threshold with the observation frame; and then updating the motion information and orientation information of the track point using the motion information and orientation information of the observation frame.

[0106] Furthermore, in some embodiments, the electronic device can also perform the above-mentioned tracking and matching process according to the sensor data type. The corresponding process is as follows: for any type of sensor data among the multiple types of sensor data, determine the tracking and matching method of the any type of sensor data according to the type of the any type of sensor data, and use the tracking and matching method of the any type of sensor data to track and match the obstacle target according to the observation box in the three-dimensional motion trajectory of the obstacle target in the any type of sensor data, so as to obtain an observation box that matches the obstacle target.

[0107] It should be noted that the specific execution of the aforementioned tracking and matching depends on the type of sensor data input to the data preprocessing module. For example, if the data preprocessing module has no millimeter-wave radar sensor data or lidar sensor data input, the processes of Mahalanobis distance matching for the millimeter-wave radar sensor data and bounding box similarity matching for the lidar sensor data will not be triggered. This achieves pluggable access to sensor data, meaning that the appropriate tracking and matching module can be adaptively selected based on the type of input sensor data.

[0108] In some embodiments, different fusion models can be used to perform the above information fusion process for different types of obstacle targets. Taking a vehicle as an example, a constant turning rate and speed model can be used for the above information fusion process. Taking a pedestrian as an example, a constant speed model can be used for the above information fusion process. Taking a roadblock as an example, a constant position model can be used for the above information fusion process.

[0109] In the above embodiments, information fusion is performed using the three-dimensional motion trajectory of the obstacle target, which enables information fusion of dynamic obstacles in both motion and orientation dimensions. This effectively reflects the motion of the obstacle target and improves the effect of information fusion.

[0110] (2-3) The electronic device determines the category of the obstacle target based on the category probability of the obstacle target in the multi-type sensor data. The category probability represents the probability that the obstacle target belongs to the corresponding category.

[0111] In this embodiment of the disclosure, the sensor data may include the category of the obstacle target and the corresponding category probability. For example, the category probability of the obstacle target may be the category probability included in fisheye camera data, the category probability included in millimeter-wave radar sensor data, or the category probability included in other sensor data. This embodiment of the disclosure does not limit the type of category probability.

[0112] In some embodiments, the electronic device selects the category with the highest category probability from the obstacle target categories in the multi-sensor data as the category of the obstacle target. For example, if the fisheye camera outputs the obstacle target category as vehicle (category probability 0.9) and the millimeter-wave radar sensor outputs the obstacle target category as human (category probability 0.3), then the obstacle target category output after category fusion is vehicle.

[0113] Alternatively, in other embodiments, the electronic device, based on the category probabilities of the obstacle target in the multi-sensor data, increments the probability score of categories with a probability greater than a preset probability by one, and then selects the category with the highest probability score as the category of the obstacle target. Here, the preset probability is a pre-set probability, such as 0.6 or 0.8. For example, if a fisheye camera outputs that the obstacle target category is "vehicle" (category probability 0.9) and a millimeter-wave radar sensor outputs that the obstacle target category is "vehicle" (category probability 0.8), it can be seen that the probability score for the category "vehicle" can be 2, and the obstacle target category output after subsequent category fusion is "vehicle".

[0114] In the above embodiments, by comprehensively analyzing the category probabilities of obstacle targets in multiple types of sensor data, the category of obstacle targets can be accurately determined, thus improving the accuracy of information fusion. It should be noted that the above embodiments only illustrate the category fusion process in two exemplary ways. In other embodiments, the electronic device can also use other methods to perform category fusion. This disclosure does not limit the method of category fusion.

[0115] (2-4) The electronic device determines whether the obstacle target exists based on the probability of the existence of the obstacle target in the multi-type sensor data. The probability of existence represents the possibility that the obstacle target exists.

[0116] The existence probability of an obstacle target is the likelihood that the obstacle target actually exists. In this embodiment, the sensor data may include the existence probability of the obstacle target. For example, the existence probability of the obstacle target may be the existence probability of a two-dimensional detection box included in the fisheye camera data, the existence probability of a three-dimensional detection box included in the LiDAR sensor data, or the existence probability of a detection box included in other sensor data. This embodiment does not limit the type of existence probability.

[0117] In some embodiments, the electronic device determines whether an obstacle exists based on the probability of its existence from the multiple types of sensor data. Alternatively, in other embodiments, the electronic device determines a probability score for the existence of an obstacle and a probability score for its non-existence based on the probability of its existence from the multiple types of sensor data, and then determines whether the obstacle exists.

[0118] In the above embodiments, by comprehensively analyzing the existence probability of obstacle targets in multiple types of sensor data, it is possible to accurately determine whether the obstacle target actually exists, thus improving the accuracy of information fusion. It should be noted that the above embodiments only illustrate the existence fusion process in two exemplary ways. In other embodiments, the electronic device can also use other methods to perform existence fusion. This disclosure does not limit the method of existence fusion.

[0119] In some embodiments, the electronic device is provided with an information fusion module, which is used to perform information fusion on the multi-type sensor data according to the target fusion algorithm.

[0120] Furthermore, in some embodiments, the information fusion module includes a static grid fusion module and a dynamic obstacle fusion module. The static grid fusion module is used to determine the occupied grid map based on the point cloud data of obstacle targets in the multi-type sensor data. The dynamic obstacle fusion module is used to perform information fusion on the multi-type sensor data based on the three-dimensional motion trajectory of the obstacle targets in the multi-type sensor data, or to determine the category of the obstacle target based on the category probability of the obstacle target in the multi-type sensor data, or to determine whether the obstacle target exists based on the existence probability of the obstacle target in the multi-type sensor data.

[0121] For example, Figure 4 This is a schematic diagram illustrating the framework of an information fusion method according to an embodiment of this disclosure. See also... Figure 4After receiving data from multiple sensors, the data preprocessing module is triggered to preprocess the data. Following this preprocessing, the information fusion module is then triggered to fuse the data. This preprocessing includes preprocessing fisheye camera point cloud data, wide-angle camera 3D bounding box data, millimeter-wave point cloud and trajectory data, LiDAR point cloud and 3D bounding box data, and ultrasonic data. After preprocessing, the data is stored in a data queue, allowing the information fusion module to extract data for fusion. This fusion process includes static grid fusion and dynamic obstacle fusion. Finally, the fused data is sent to the downstream driving planning module and human-machine interface rendering module. The driving planning module uses the fused data for driving planning, and the human-machine interface rendering module uses it to render the interface, displaying the surrounding road environment of the autonomous vehicle to the user on the in-vehicle terminal.

[0122] For example, Figure 5 This is a schematic diagram illustrating a dynamic obstacle fusion process according to an embodiment of this disclosure. See also... Figure 5 Dynamic obstacle fusion comprises two parts: association matching (i.e., the tracking matching process described above) and information fusion. Association matching can include front fisheye data cascade matching, front wide-angle camera 3D bounding box similarity matching, millimeter-wave radar track point Mahalanobis distance matching, lidar 3D bounding box similarity matching, and ultrasonic point geometric similarity matching.

[0123] Information fusion includes motion and orientation fusion, category-based probability fusion, and existence-based probability fusion. Motion and orientation fusion includes the fusion of 3D track (i.e., 3D motion trajectory) and various sensor data, such as fusion of 3D track with fisheye camera point clouds, fusion of 3D track with fisheye camera 2D bounding boxes, fusion of 3D track with wide-angle camera 3D bounding boxes, fusion of 3D track with millimeter-wave track points, fusion of 3D track with LiDAR 3D bounding boxes, and fusion of 3D track with ultrasonic points. Similarly, when the data preprocessing module has no input of millimeter-wave radar or LiDAR sensor data, the information fusion process for millimeter-wave radar and LiDAR will not be triggered. This achieves pluggable access to sensor data.

[0124] In some embodiments, before implementing this solution, technicians will design and compile configuration files for the information fusion module of the electronic device. Specifically, in the compilation configuration file of the information fusion module, different identifier fields are set for different fusion computing power platforms, different compilation options are set for different fusion computing power platforms, and different status flags are set for different fusion computing power platforms, etc.

[0125] For example, assuming the sensor combination of fusion computing platform A is a fisheye camera and a wide-angle camera, then the fusion algorithm corresponding to fusion computing platform A is algorithm scheme a, which performs information fusion between the fisheye camera and the wide-angle camera. Alternatively, assuming the sensor combination of fusion computing platform B is a fisheye camera and an ultrasonic radar sensor, then the fusion algorithm corresponding to fusion computing platform B is algorithm scheme b, which performs information fusion between the fisheye camera and the ultrasonic radar sensor.

[0126] The above steps S301 to S304 illustrate the solution by first preprocessing data from multiple types of sensors, and then using the determined target fusion algorithm to fuse the preprocessed data. It should be noted that S301 is an optional step. In some embodiments, the electronic device does not need to execute S301; correspondingly, S304 can be replaced by: upon receiving the multi-type sensor data from the autonomous vehicle, the electronic device performs information fusion on the multi-type sensor data of the autonomous vehicle according to the target fusion algorithm to obtain the fused target sensor data.

[0127] In this way, for different types of autonomous vehicles, a fusion algorithm suitable for that type of autonomous vehicle can be determined based on the type of fusion computing platform. This algorithm is compatible with different fusion computing platforms and different sensor combinations, effectively solving the information fusion problem of different sensor combinations and different fusion computing platforms. It enables autonomous driving applications that are compatible with multiple vehicle models with a single codebase, thereby effectively reducing the maintenance cost of information fusion and greatly improving the system's resource utilization.

[0128] The technical solution provided in this disclosure can determine the target fusion algorithm corresponding to the fusion computing platform running the autonomous vehicle by obtaining the system environmental variables of the autonomous vehicle. Since different fusion computing platforms correspond to different sensor combinations and different fusion algorithms, the target fusion algorithm determined according to the system environmental variables of the autonomous vehicle is a fusion algorithm that is compatible with the current autonomous vehicle. That is, it is a fusion algorithm that can be applied to multiple types of sensors in the current autonomous vehicle. Therefore, when receiving multiple types of sensor data from the autonomous vehicle, information fusion is performed on the multiple types of sensor data according to the target fusion algorithm, which can effectively improve the accuracy of information fusion.

[0129] Figure 6 This is a structural block diagram of an information fusion device according to an embodiment of this disclosure. See also... Figure 6 The device includes an acquisition module 601, a determination module 602, and a fusion module 603. Wherein:

[0130] The acquisition module 601 is used to acquire the system environment variables of the autonomous vehicle. The system environment variables are used to indicate the system environment of the fusion computing platform on which the autonomous vehicle is running. Different fusion computing platforms correspond to different sensor combinations, and the sensor combinations correspond to multiple types of sensors.

[0131] The determination module 602 is used to determine the target fusion algorithm corresponding to the fusion computing platform based on the system environmental variables of the autonomous vehicle. The target fusion algorithm is used to perform information fusion based on the sensor combination corresponding to the fusion computing platform; different fusion computing platforms correspond to different fusion algorithms.

[0132] The fusion module 603 is used to perform information fusion on the multi-type sensor data received from the autonomous vehicle according to the target fusion algorithm to obtain the fused target sensor data, which is used for autonomous driving operations.

[0133] The technical solution provided in this disclosure can determine the target fusion algorithm corresponding to the fusion computing platform running the autonomous vehicle by obtaining the system environmental variables of the autonomous vehicle. Since different fusion computing platforms correspond to different sensor combinations and different fusion algorithms, the target fusion algorithm determined according to the system environmental variables of the autonomous vehicle is a fusion algorithm that is compatible with the current autonomous vehicle. That is, it is a fusion algorithm that can be applied to multiple types of sensors in the current autonomous vehicle. Therefore, when receiving multiple types of sensor data from the autonomous vehicle, information fusion is performed on the multiple types of sensor data according to the target fusion algorithm, which can effectively improve the accuracy of information fusion.

[0134] In some embodiments, the determining module 602 includes:

[0135] The determination submodule is used to determine the fusion computing platform running on the autonomous vehicle based on the system environment variables of the autonomous vehicle.

[0136] The acquisition submodule is used to obtain the algorithm configuration information corresponding to the identification information of the fusion computing platform running the autonomous vehicle from a preset correspondence, and use it as the target fusion algorithm. The preset correspondence includes multiple identification information and the algorithm configuration information corresponding to the multiple identification information.

[0137] In some embodiments, the determining submodule is configured to:

[0138] Based on the system environment variables of the autonomous vehicle, determine the state flag quantity corresponding to the system environment variables. This state flag quantity is used to indicate the type of the fusion computing platform.

[0139] The fusion computing platform running on the autonomous vehicle is determined based on the state flags corresponding to the system's environmental variables.

[0140] In some embodiments, a switching module is further included, for:

[0141] When the system's environmental variables change, in response to the state flag quantity switching from the original flag quantity to the target flag quantity, the target fusion algorithm is switched to the fusion algorithm corresponding to the target flag quantity.

[0142] In some embodiments, the fusion module 603 is configured to:

[0143] Determine the target compilation options for the target fusion algorithm. These target compilation options are matched with the sensor combination corresponding to the fusion computing platform. One of the compilation options in the target compilation options corresponds to the fusion sub-algorithm of a type of sensor in the sensor combination. Different fusion computing platforms correspond to different compilation options.

[0144] According to the fusion sub-algorithm corresponding to the target compilation option, information fusion is performed on the data from the multiple types of sensors.

[0145] In some embodiments, the fusion module 603 includes a fusion submodule, comprising at least one of the following:

[0146] The category fusion submodule is used to determine the category of the obstacle target based on the category probability of the obstacle target in the multi-class sensor data. The category probability represents the probability that the obstacle target belongs to the corresponding category.

[0147] The existence fusion submodule is used to determine whether an obstacle target exists based on the existence probability of the obstacle target in the multi-type sensor data. The existence probability represents the likelihood that the obstacle target exists.

[0148] The information fusion submodule is used to perform information fusion on the multi-sensor data based on the three-dimensional motion trajectory of the obstacle target in the multi-sensor data. The three-dimensional motion trajectory is used to present the motion trajectory of the obstacle target in the three-dimensional image.

[0149] The map fusion submodule is used to determine the occupied grid map based on the point cloud data of obstacle targets in the multi-type sensor data. The occupied grid map is used to present the distribution of multiple obstacle targets in the form of a grid map.

[0150] In some embodiments, the information fusion submodule includes:

[0151] The tracking and matching submodule is used to track and match the obstacle target based on the observation box in the three-dimensional motion trajectory of the obstacle target in the multi-type sensor data, and obtain the observation box that matches the obstacle target.

[0152] The information update submodule is used to update the motion and orientation information of the obstacle target using Kalman filtering based on the observation box that matches the obstacle target.

[0153] In some embodiments, the tracking and matching submodule is configured to:

[0154] For any one type of sensor data among the multiple types of sensor data, determine the tracking and matching method for that type of sensor data based on the type of that type of sensor data;

[0155] Based on the observation box in the three-dimensional motion trajectory of the obstacle target in any type of sensor data, the tracking and matching method of any type of sensor data is used to track and match the obstacle target to obtain the observation box that matches the obstacle target.

[0156] In some embodiments, the tracking matching includes at least one of the following: cascade matching of fisheye camera data, detection box similarity matching of wide-angle camera data, Mahalanobis distance matching of millimeter-wave radar sensor data, detection box similarity matching of lidar sensor data, and geometric similarity matching of ultrasonic radar sensor data.

[0157] In some embodiments, a processing module is further included, configured to preprocess any type of sensor data received from the plurality of sensor data to obtain the preprocessed plurality of sensor data.

[0158] This fusion module is used to perform information fusion on the preprocessed multi-type sensor data according to the target fusion algorithm.

[0159] In some embodiments, the processing module is used for at least one of the following:

[0160] Perform outlier filtering on the point cloud data of obstacle targets in any type of sensor data;

[0161] The bounding box of the obstacle target is obtained by fitting the bounding box to the point cloud data of the obstacle target in the data of any type of sensor.

[0162] Based on the two-dimensional detection bounding box of the obstacle target in the data of any type of sensor, determine the orientation of the obstacle target;

[0163] Kalman filtering is applied to the 3D detection bounding box of the obstacle target in the sensor data of any type to track the velocity of the obstacle target;

[0164] Based on the point cloud data and trajectory data of the obstacle target in any type of sensor data, determine the location, category, and size of the obstacle target.

[0165] According to embodiments of the present disclosure, the present disclosure also provides an electronic device, including at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the information fusion method provided by the present disclosure.

[0166] According to embodiments of this disclosure, this disclosure also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause an electronic device to execute the information fusion method provided in this disclosure.

[0167] According to embodiments of this disclosure, this disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the information fusion method provided in this disclosure.

[0168] In some embodiments, the electronic device may be as described above. Figure 1 The vehicle-mounted equipment shown. Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. Electronic device 700 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic device 700 may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0169] like Figure 7As shown, the electronic device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. The RAM 703 may also store various programs and data required for the operation of the electronic device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0170] Multiple components in electronic device 700 are connected to I / O interface 705, including: input unit 706, such as keyboard, mouse, etc.; output unit 707, such as various types of displays, speakers, etc.; storage unit 708, such as disk, optical disk, etc.; and communication unit 709, such as network card, modem, wireless transceiver, etc. Communication unit 709 allows electronic device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0171] The computing unit 701 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as information fusion methods. For example, in some embodiments, the information fusion method can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the information fusion method described above can be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform the information fusion method by any other suitable means (e.g., by means of firmware).

[0172] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard parts (ASSPs), systems-on-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0173] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0174] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory (EPROM or flash memory), optical fibers, compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0175] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user, such as a cathode ray tube (CRT) or liquid crystal display (LCD) monitor; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0176] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0177] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0178] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this disclosure can be achieved, and this is not limited herein.

[0179] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. An information fusion method, comprising: The system environment variables of the autonomous vehicle are obtained, which are used to indicate the system environment of the fusion computing platform on which the autonomous vehicle is running; different fusion computing platforms correspond to different sensor combinations, and the sensor combinations correspond to multiple types of sensors; Based on the system environment variables of the autonomous vehicle, the fusion computing platform on which the autonomous vehicle is running is determined, and the target fusion algorithm corresponding to the fusion computing platform is determined. The target fusion algorithm is used to perform information fusion based on the sensor combination corresponding to the fusion computing platform; different fusion computing platforms correspond to different fusion algorithms. Upon receiving multi-type sensor data from the autonomous vehicle, the multi-type sensor data is fused according to the target fusion algorithm to obtain fused target sensor data, which is used for autonomous driving operations.

2. The method according to claim 1, wherein, The determination of the target fusion algorithm corresponding to the fusion computing platform includes: Based on the identification information corresponding to the fusion computing platform running the autonomous vehicle, algorithm configuration information corresponding to the identification information is obtained from a preset correspondence and used as the target fusion algorithm. The preset correspondence includes multiple identification information and the algorithm configuration information corresponding to the multiple identification information.

3. The method according to claim 1, wherein, The step of determining the fusion computing platform running on the autonomous vehicle based on the system environment variables of the autonomous vehicle includes: Based on the system environment variables of the autonomous vehicle, a state flag quantity corresponding to the system environment variables is determined, and the state flag quantity is used to indicate the type of the fusion computing power platform; The fusion computing platform running on the autonomous vehicle is determined based on the state flags corresponding to the system environment variables.

4. The method according to claim 3, further comprising: When the system environment variables change, in response to the state flag quantity switching from the original flag quantity to the target flag quantity, the target fusion algorithm is switched to the fusion algorithm corresponding to the target flag quantity.

5. The method according to claim 1, wherein, The step of fusing information from the multiple types of sensor data according to the target fusion algorithm includes: Determine the target compilation options for the target fusion algorithm. The target compilation options are matched with the sensor combination corresponding to the fusion computing platform. One of the compilation options corresponds to the fusion sub-algorithm of a type of sensor in the sensor combination. Different fusion computing platforms correspond to different compilation options. Information fusion is performed on the multi-type sensor data according to the fusion sub-algorithm corresponding to the target compilation options.

6. The method according to claim 1 or 5, wherein, The information fusion of the multiple types of sensor data includes at least one of the following: The category of the obstacle target is determined based on the category probability of the obstacle target in the multi-type sensor data, where the category probability represents the likelihood that the obstacle target belongs to the corresponding category; Based on the probability of the existence of the obstacle target in the multi-type sensor data, it is determined whether the obstacle target exists, and the probability of existence represents the likelihood of the obstacle target existing. Based on the three-dimensional motion trajectory of the obstacle target in the multi-type sensor data, information fusion is performed on the multi-type sensor data, and the three-dimensional motion trajectory is used to present the motion trajectory of the obstacle target in a three-dimensional image; Based on the point cloud data of obstacle targets in the multi-type sensor data, an occupied grid map is determined. The occupied grid map is used to present the distribution of multiple obstacle targets in the form of a grid map.

7. The method according to claim 6, wherein, The step of fusing information from the multi-type sensor data based on the three-dimensional motion trajectory of the obstacle target in the multi-type sensor data includes: Based on the observation boxes in the three-dimensional motion trajectory of the obstacle target in the multi-type sensor data, the obstacle target is tracked and matched to obtain the observation box that matches the obstacle target; Based on the observation frame that matches the obstacle target, the motion and orientation information of the obstacle target are updated using Kalman filtering.

8. The method according to claim 7, wherein, The step of tracking and matching the obstacle target based on the observation box in the three-dimensional motion trajectory of the obstacle target in the multi-type sensor data to obtain the observation box that matches the obstacle target includes: For any one type of sensor data among the multiple types of sensor data, determine the tracking and matching method for that type of sensor data based on the type of that type of sensor data; Based on the observation box in the three-dimensional motion trajectory of the obstacle target in any type of sensor data, the obstacle target is tracked and matched using the tracking and matching method of the sensor data of any type of sensor data to obtain the observation box that matches the obstacle target.

9. The method according to claim 7 or 8, wherein, The tracking matching includes at least one of the following: cascade matching of fisheye camera data, detection box similarity matching of wide-angle camera data, Mahalanobis distance matching of millimeter-wave radar sensor data, detection box similarity matching of lidar sensor data, and geometric similarity matching of ultrasonic radar sensor data.

10. The method according to claim 1, further comprising: In response to receiving any type of sensor data from the multiple types of sensor data, data preprocessing is performed on the data of any type of sensor data to obtain the multiple types of sensor data after data preprocessing. The step of fusing information from the multiple types of sensor data according to the target fusion algorithm includes: According to the target fusion algorithm, information fusion is performed on the preprocessed multi-type sensor data.

11. The method according to claim 10, wherein, The data preprocessing for any of the aforementioned sensor data includes at least one of the following: Anomaly filtering is performed on the point cloud data of obstacle targets in any of the aforementioned sensor data; The bounding box of the obstacle target is obtained by fitting the bounding box of the obstacle target based on the point cloud data of the obstacle target in any of the sensor data. Based on the two-dimensional detection bounding box of the obstacle target in any type of sensor data, determine the orientation of the obstacle target; Kalman filtering is applied to the three-dimensional detection bounding box of the obstacle target in any type of sensor data to track the velocity of the obstacle target; Based on the point cloud data and trajectory data of the obstacle target in any of the sensor data, determine the position, category, and size of the obstacle target.

12. An information fusion device, comprising: The acquisition module is used to acquire system environment variables of the autonomous vehicle. The system environment variables are used to indicate the system environment of the fusion computing platform on which the autonomous vehicle is running. Different fusion computing platforms correspond to different sensor combinations, and the sensor combinations correspond to multiple types of sensors. The determination module is used to determine the fusion computing platform running by the autonomous vehicle based on the system environment variables of the autonomous vehicle, and to determine the target fusion algorithm corresponding to the fusion computing platform. The target fusion algorithm is used to perform information fusion based on the sensor combination corresponding to the fusion computing platform; different fusion computing platforms correspond to different fusion algorithms. The fusion module is used to perform information fusion on the multi-type sensor data received from the autonomous vehicle according to the target fusion algorithm to obtain the fused target sensor data, which is used for autonomous driving operations.

13. The apparatus according to claim 12, wherein, The determining module includes: The acquisition submodule is used to acquire algorithm configuration information corresponding to the identification information of the fusion computing platform running the autonomous vehicle from a preset correspondence, based on the identification information corresponding to the identification information, as the target fusion algorithm. The preset correspondence includes multiple identification information and the algorithm configuration information corresponding to the multiple identification information.

14. The apparatus according to claim 12, wherein, The determining module includes: The determination submodule is used to determine the state flag quantity corresponding to the system environment variables of the autonomous vehicle, and the state flag quantity is used to indicate the type of the fusion computing power platform; The fusion computing platform running on the autonomous vehicle is determined based on the state flags corresponding to the system environment variables.

15. The apparatus of claim 14, further comprising a switching module for: When the system environment variables change, in response to the state flag quantity switching from the original flag quantity to the target flag quantity, the target fusion algorithm is switched to the fusion algorithm corresponding to the target flag quantity.

16. The apparatus according to claim 12, wherein, The fusion module is used for: Determine the target compilation options for the target fusion algorithm. The target compilation options are matched with the sensor combination corresponding to the fusion computing platform. One of the compilation options corresponds to the fusion sub-algorithm of a type of sensor in the sensor combination. Different fusion computing platforms correspond to different compilation options. Information fusion is performed on the multi-type sensor data according to the fusion sub-algorithm corresponding to the target compilation options.

17. The apparatus according to claim 12 or 16, wherein, The fusion module includes a fusion submodule, which includes at least one of the following: The category fusion submodule is used to determine the category of the obstacle target based on the category probability of the obstacle target in the multi-category sensor data, wherein the category probability represents the probability that the obstacle target belongs to the corresponding category; The existence fusion submodule is used to determine whether the obstacle target exists based on the existence probability of the obstacle target in the multi-type sensor data, wherein the existence probability represents the possibility that the obstacle target exists; The information fusion submodule is used to perform information fusion on the multi-type sensor data based on the three-dimensional motion trajectory of the obstacle target in the multi-type sensor data, and the three-dimensional motion trajectory is used to present the motion trajectory of the obstacle target in a three-dimensional image; The map fusion submodule is used to determine the occupied grid map based on the point cloud data of obstacle targets in the multi-type sensor data. The occupied grid map is used to present the distribution of multiple obstacle targets in the form of a grid map.

18. The apparatus according to claim 17, wherein, The information fusion submodule includes: The tracking and matching submodule is used to track and match the obstacle target based on the observation box in the three-dimensional motion trajectory of the obstacle target in the multi-type sensor data, so as to obtain the observation box that matches the obstacle target; The information update submodule is used to update the motion and orientation information of the obstacle target using Kalman filtering based on the observation frame that matches the obstacle target.

19. The apparatus according to claim 18, wherein, The tracking and matching submodule is used for: For any one type of sensor data among the multiple types of sensor data, determine the tracking and matching method for that type of sensor data based on the type of that type of sensor data; Based on the observation box in the three-dimensional motion trajectory of the obstacle target in any type of sensor data, the obstacle target is tracked and matched using the tracking and matching method of the sensor data of any type of sensor data to obtain the observation box that matches the obstacle target.

20. The apparatus according to claim 18 or 19, wherein, The tracking matching includes at least one of the following: cascade matching of fisheye camera data, detection box similarity matching of wide-angle camera data, Mahalanobis distance matching of millimeter-wave radar sensor data, detection box similarity matching of lidar sensor data, and geometric similarity matching of ultrasonic radar sensor data.

21. The apparatus according to claim 12, further comprising a processing module, configured to, in response to receiving any type of sensor data from the plurality of sensor data, perform data preprocessing on the any type of sensor data to obtain the data preprocessed from the plurality of sensor data; The fusion module is used to perform information fusion on the preprocessed multi-type sensor data according to the target fusion algorithm.

22. The apparatus according to claim 21, wherein, The processing module is used for at least one of the following: Anomaly filtering is performed on the point cloud data of obstacle targets in any of the aforementioned sensor data; The bounding box of the obstacle target is obtained by fitting the bounding box of the obstacle target based on the point cloud data of the obstacle target in any of the sensor data. Based on the two-dimensional detection bounding box of the obstacle target in any type of sensor data, determine the orientation of the obstacle target; Kalman filtering is applied to the three-dimensional detection bounding box of the obstacle target in any type of sensor data to track the velocity of the obstacle target; Based on the point cloud data and trajectory data of the obstacle target in any of the sensor data, determine the position, category, and size of the obstacle target.

23. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 11.

24. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the electronic device to perform the method according to any one of claims 1 to 11.

25. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 11.

Images