Multi-sensor fusion environment sensing method and device, electronic equipment and medium

Abstract

The invention discloses a multi-sensor fusion environment sensing method and device, electronic equipment and a medium. The method comprises the following steps: acquiring multi-dimensional environment data of a target environment collected by a plurality of preset sensors; identifying a key target in the target environment based on the multi-dimensional environment data and target features of the key target; obtaining weight data corresponding to each sensor in the plurality of sensors; performing fusion processing on the multi-dimensional environment data based on weight data corresponding to the plurality of sensors and the target features to obtain fusion data and fusion feature information therein, the weight data including a weight corresponding to each sensor; constructing a three-dimensional environment map of the target environment based on the fusion data; and determining environment perception information of the target environment based on the three-dimensional environment map and the fusion feature information. According to the invention, the precision and robustness of environmental perception can be improved, errors and redundancy are reduced, and the reliability of a result is improved.

Description

Technical Field

[0001] The present application relates to the field of intelligent sensing technology, and in particular to a multi-sensor fusion environment perception method and device, electronic equipment, and medium. Background Art

[0002] In today's technological application field, environmental perception systems play a vital role in the development of many industries. Fields such as intelligent driving, security monitoring, and industrial automation are highly dependent on accurate and reliable environmental perception to achieve normal operation of the system.

[0003] Most of the mainstream environmental perception systems currently on the market rely on a single sensor or simple data overlay technology. These systems face many challenges in practical applications. For example, some systems rely solely on cameras for target detection and positioning. However, this approach can significantly reduce perception accuracy and severely affect robustness (i.e., the robustness and reliability of the system) in complex environments such as insufficient light, rainy or snowy weather, or when the target is obscured. On the other hand, even if a simple data overlay method is used to integrate information from multiple sensors, this method often fails to fully utilize the unique advantages of each sensor. Because simple overlay cannot compensate for the errors and limitations of each sensor in real time, the system is still prone to problems such as misjudgment and missed detection when faced with complex and changing real-world scenarios. Summary of the Invention

[0004] In order to solve the above problems, embodiments of the present application provide a multi-sensor fusion environment perception method, device, electronic device, computer-readable storage medium and computer program product.

[0005] First, in order to solve the above technical problems, the present application provides an environment perception method using multi-sensor fusion, including:

[0006] Acquire multi-dimensional environmental data of the target environment, wherein the multi-dimensional environmental data is environmental data of different dimensions collected by a plurality of preset sensors;

[0007] Performing target recognition based on the multi-dimensional environmental data to determine key targets within the target environment and target characteristics of the key targets;

[0008] Based on weight data corresponding to the multiple sensors and the target feature, the multi-dimensional environmental data is fused to obtain fused data and fused feature information corresponding to the target feature in the fused data, wherein the weight data includes a weight corresponding to each sensor in the multiple sensors;

[0009] constructing a three-dimensional environment map of the target environment based on the fused data;

[0010] Environmental perception information of the target environment is determined based on the three-dimensional environment map and the fused feature information.

[0011] The beneficial effects are:

[0012] In the technical solution provided in the embodiment of the present application, the key targets in the target environment are identified by acquiring the multi-dimensional environmental data of the target environment transmitted by multiple sensors, and the target features of the key targets are extracted based on the multi-dimensional environmental data; then the weight data corresponding to each sensor in the multiple sensors is acquired; and the multi-dimensional environmental data is fused based on the weight data and the target features to obtain the fused data and the fused feature information corresponding to the target features in the fused data; and then a three-dimensional environmental map of the target environment is constructed based on the fused data, thereby determining the environmental perception information of the target environment based on the three-dimensional environmental map and the fused feature information. In this way, the present application can process multi-source information in real time by fusing data from multiple sensors, and can effectively overcome the limitations of a single sensor in a complex environment, achieving more accurate and stable environmental perception. In addition, the multi-dimensional environmental data can be fused by the weight data of each sensor to compensate for the error of a single sensor, thereby improving the accuracy and robustness of environmental perception, reducing errors and redundancy, and improving the reliability of the overall system.

[0013] In a second aspect, the present invention provides an environment perception device for multi-sensor fusion, comprising an acquisition unit, a recognition unit, a fusion unit, a map construction unit, and a perception unit;

[0014] An acquisition unit, configured to acquire multi-dimensional environmental data of a target environment, wherein the multi-dimensional environmental data is environmental data of different dimensions collected by a plurality of preset sensors;

[0015] an identification unit, configured to perform target identification based on the multi-dimensional environmental data, and determine key targets within the target environment, as well as target features of the key targets;

[0016] a fusion unit, configured to perform a fusion process on the multi-dimensional environmental data based on weight data corresponding to the multiple sensors and the target feature, to obtain fused data and fused feature information corresponding to the target feature in the fused data, wherein the weight data includes a weight corresponding to each sensor of the multiple sensors;

[0017] A map construction unit, configured to construct a three-dimensional environment map of the target environment based on the fused data;

[0018] A perception unit is used to determine environmental perception information of the target environment based on the three-dimensional environment map and the fused feature information.

[0019] In a third aspect, the present application also provides an electronic device comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the electronic device implements the multi-sensor fusion environmental perception method as described above.

[0020] In a fourth aspect, the present application also provides a computer-readable storage medium having computer-readable instructions stored thereon. When the computer-readable instructions are executed by a processor of a computer, the computer executes the multi-sensor fusion environment perception method as described above.

[0021] In a fifth aspect, the present application further provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the multi-sensor fusion environment perception method provided in the various optional embodiments described above.

[0022] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings are incorporated into and constitute a part of the specification, illustrating embodiments consistent with the present application and, together with the specification, serving to explain the principles of the present application. It is obvious that the drawings described below are merely some embodiments of the present application, and a person of ordinary skill in the art can derive other drawings based on these drawings without inventive effort. In the drawings:

[0024] Figure 1 This is a flowchart of an environmental perception method of multi-sensor fusion shown in an exemplary embodiment of the present application;

[0025] Figure 2 This is a block diagram of an environment perception device with multi-sensor fusion, shown in an exemplary embodiment of the present application;

[0026] Figure 3 It is a structural diagram of a computer system suitable for implementing the electronic device of the embodiment of the present application. DETAILED DESCRIPTION

[0027] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. When the following description refers to the drawings, identical numerals in different figures represent identical or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0028] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically separate entities. That is, these functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0029] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, while others may be combined or partially combined. Therefore, the actual execution order may vary depending on the actual situation.

[0030] In this application, "plurality" refers to two or more. "And / or" describes the relationship between related objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, and B exists alone. The character " / " generally indicates that the related objects are in an "or" relationship.

[0031] In this application, all actions related to the acquisition of signals, information or data are carried out in strict compliance with the relevant data protection laws and policies of the country where they are located, and with the authorization of the owner of the corresponding device.

[0032] Owner refers to the person or entity that owns or controls the relevant device (which may be a device, system or other tool that can collect data).

[0033] In the field of intelligent connected vehicles, “owners” mainly include:

[0034] (1) Automobile manufacturers: As vehicle hardware and system developers, they control the underlying hardware and software platforms of the vehicle and have the right to manage and control the data generated by vehicle operation, such as driving and fault data.

[0035] (2) Parts suppliers: They provide key components for automobiles and have certain ownership of the data collected and processed by the components, which is used for product optimization and after-sales service, such as data generated by sensors and chips.

[0036] (3) Vehicle owner or user: The actual user of the vehicle, who has the right to decide how and to what extent vehicle data is used, such as whether to share data such as driving trajectory and driving habits, and has the need and right to protect the privacy of his or her own relevant data.

[0037] (4) Service providers: provide software, data analysis and other services, and have the right to use and manage the acquired and processed data within the framework of the agreement, but the ownership usually belongs to other entities.

[0038] In order to solve the problem that most current environmental perception systems rely on a single sensor or simple data superposition technology, resulting in insufficient perception accuracy and prone to misjudgment, missed detection, etc., the embodiments of the present application propose a multi-sensor fusion environmental perception method and device, electronic device, and computer-readable storage medium, which mainly involve the multi-sensor fusion environmental perception technology included in intelligent perception technology. These embodiments will be described in detail below.

[0039] First see Figure 1 , Figure 1 This is a flowchart of an environmental perception method using multi-sensor fusion, as illustrated by an exemplary embodiment of the present application. The method can be specifically executed by a server, which can be a standalone server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, without limitation herein.

[0040] like Figure 1 As shown, in an exemplary embodiment, the multi-sensor fusion environment perception method may include steps S101 to S105, which are described in detail as follows:

[0041] Step S101 : acquiring multi-dimensional environmental data of a target environment. The multi-dimensional environmental data is environmental data of different dimensions collected by a plurality of preset sensors.

[0042] Step S102 : performing target recognition based on the multi-dimensional environmental data to determine key targets in the target environment and target features of the key targets.

[0043] In step S103, based on the weight data and target features corresponding to the multiple sensors, the multi-dimensional environmental data is fused to obtain fused data and fused feature information corresponding to the target features in the fused data. The weight data includes a weight corresponding to each sensor in the multiple sensors.

[0044] Step S104: constructing a three-dimensional environment map of the target environment based on the fused data.

[0045] Step S105 : determining environmental perception information of the target environment based on the three-dimensional environment map and the fused feature information.

[0046] As can be seen from the above, in the method provided by this embodiment, the key targets in the target environment are identified by acquiring the multi-dimensional environmental data of the target environment transmitted by multiple sensors, and the target features of the key targets are extracted based on the multi-dimensional environmental data; then the weight data corresponding to each sensor in the multiple sensors is acquired; and the multi-dimensional environmental data is fused based on the weight data and the target features to obtain the fused data and the fused feature information corresponding to the target features in the fused data; then a three-dimensional environmental map of the target environment is constructed based on the fused data, and thus the environmental perception information of the target environment is determined based on the three-dimensional environmental map and the fused feature information. In this way, on the one hand, the present application can process multi-source information in real time by fusing multiple sensor data, and can effectively overcome the limitations of a single sensor in a complex environment, and achieve more accurate and stable environmental perception; on the other hand, by fusing the multi-dimensional environmental data through the weight data of each sensor, the error of a single sensor can be compensated, thereby improving the accuracy and robustness of environmental perception, reducing errors and redundancy, and improving the reliability of the overall system.

[0047] The purpose of acquiring environmental perception information in the embodiments of the present application is to accurately understand the environment and optimize decision-making through real-time data collection and analysis. For example, in the target scenario of autonomous decision-making by intelligent systems, environmental perception information can be used to perceive road conditions, pedestrians, and obstacles, and to achieve real-time path planning and obstacle avoidance. Industrial robots can adjust their grasping force or movement trajectory through environmental perception to adapt to the needs of flexible production lines. In the target scenario corresponding to resource management and efficiency optimization, environmental perception information can be combined with traffic flow and weather information to plan the optimal path during logistics and transportation, reducing fuel consumption and delay rate. When applied to energy systems, environmental perception information can be used to achieve dynamic supply and demand balance. In the target scenario corresponding to scientific research and ecological protection, environmental perception information can be used to identify and track wildlife activities and assess ecological balance.

[0048] In an exemplary embodiment provided by the present application, target features include shape features, position features, orientation features, and speed features. The specific steps of fusing multi-dimensional environmental data based on weight data and target features corresponding to multiple sensors to obtain fused data and fused feature information corresponding to the target features in the fused data may include:

[0049] Using a Kalman filter fusion algorithm, the dynamic data corresponding to the position feature and the speed feature in the multi-dimensional environmental data are fused based on the weight data to obtain first fused feature data;

[0050] Using a particle filter fusion algorithm, the orientation data corresponding to the orientation feature in the multi-dimensional environmental data is fused based on the weight data to obtain second fused feature data;

[0051] Using a deep learning fusion algorithm, the static data corresponding to the shape features in the multi-dimensional environmental data are fused based on the weight data to obtain third fused feature data;

[0052] Acquire other data except the dynamic data, the orientation data, and the static data from the multi-dimensional environmental data, and fuse the other data based on the weight data to obtain fourth fused data;

[0053] The first fused feature data, the second fused data and the third fused data are used as fused feature information corresponding to the target feature;

[0054] The fused data of the multi-dimensional environment data is obtained based on the fused feature information and the fourth fused data.

[0055] In this embodiment, the data included in the multidimensional environmental data is fused separately according to data type. For the dynamic data corresponding to the position and velocity features, a Kalman filter fusion algorithm is used to obtain first fused feature data. For the orientation data corresponding to the orientation feature, a particle filter fusion algorithm is used to obtain second fused feature data. For the static data corresponding to the shape feature, a deep learning fusion algorithm is used to obtain third fused feature data. The first, second, and third fused feature data are then used as fused feature information corresponding to the target feature. For example, the fused feature information corresponding to a key target can be expressed as: category = "truck", position = (x, y, z), speed = 20 m / s, and orientation = 30°. For all data in the multidimensional environmental data other than the dynamic, orientation, and static data, a simple data fusion is performed based on the weight data corresponding to multiple sensors to obtain fourth fused data. Fusion data of the multidimensional environmental data is then obtained based on the fused feature information and the fourth fused data.

[0056] In addition, in order to ensure the accuracy and quality of the fused data, spatiotemporal alignment processing is required before data fusion. Preferably, the multi-dimensional environmental data is synchronized by linear interpolation according to the timestamp, and all data are uniformly mapped to the same spatiotemporal coordinate system (such as the vehicle coordinate system or the map coordinate system) through the spatial transformation matrix.

[0057] In this way, through the above embodiments, this application effectively integrates laser, vision, radar and IMU data based on a data fusion algorithm that combines Kalman filtering and deep learning to achieve high-precision environmental modeling.

[0058] In an exemplary embodiment provided by this application, the specific steps of obtaining multi-dimensional environmental data of a target environment transmitted by a plurality of preset sensors may include:

[0059] Perform consistency processing on multiple preset sensors to obtain multiple processed sensors. The consistency processing includes acquisition time synchronization and parameter calibration.

[0060] The multi-dimensional environmental data of the target environment is obtained through the processed multiple sensors, and the multi-dimensional environmental data is subjected to noise filtering.

[0061] In this embodiment, the preset multiple sensors may include lidar, millimeter-wave radar, high-definition camera, ultrasonic sensor and inertial measurement unit (IMU), which can collect lidar data, millimeter-wave radar data, image data, ultrasonic data and motion status information in the target environment.

[0062] Before using multiple sensors to acquire multi-dimensional environmental data, to ensure the accuracy and consistency of multi-source data, multiple sensors are processed through consistency processing. This consistency processing includes synchronizing acquisition times and calibrating parameters to obtain the processed multiple sensors. Specifically, consistency processing involves calibrating the acquisition times from each sensor using hardware triggers or software algorithms to ensure data is processed using the same time reference. It also calibrates the internal and external parameters of each sensor to eliminate installation errors and system deviations, ensuring data spatial uniformity.

[0063] After obtaining the multi-dimensional environmental data of the target environment through the processed multiple sensors, the multi-dimensional environmental data is subjected to noise filtering, such as using low-pass filtering, mean filtering or Kalman filtering to suppress noise and improve signal quality.

[0064] In this way, through the above embodiments, the present application improves the accuracy and effectiveness of multi-dimensional environmental data by performing consistency processing on the sensor before data collection and filtering the data after collection.

[0065] In an exemplary embodiment provided by this application, the multi-dimensional environmental data includes image data. Therefore, the specific steps of performing target recognition based on the multi-dimensional environmental data and determining key targets within the target environment and target characteristics of the key targets may include:

[0066] Based on the preset convolutional layer network, hierarchical features of image data are extracted to obtain environmental hierarchical features;

[0067] Identify the object to be tested in the target environment based on the edge features of the environment through a preset target recognition network;

[0068] Perform semantic segmentation on the object to be tested to obtain the semantic category of the object to be tested;

[0069] Perform object segmentation processing on the object to be tested, and identify key targets based on semantic categories from the processed object to be tested. The object segmentation processing includes instance segmentation and edge enhancement;

[0070] The target features of key targets are extracted based on multi-dimensional environmental data.

[0071] In this embodiment, the preset convolutional layer network can be a multi-layer CNN (Convolutional Neural Network), a convolutional neural network, and the preset target recognition network can be a YOLO (You Only Look Once) target detection algorithm.

[0072] The preferred process for identifying key targets is to first use a multi-layer CNN to extract low-level edge features (such as Sobel edges), mid-level texture information, and high-level semantic features from the image data as environmental layer features. Edges are areas of sudden brightness changes in an image, and texture is a repetitive pattern in a local area of ​​the image. It lies between low-level edge features and high-level semantic features. High-level semantic features represent more abstract and conceptual information in the image, such as the category of objects in the image or the theme of the scene.

[0073] The object detection network YOLO is then used to identify objects within the target environment, such as pedestrians, vehicles, and traffic signs, based on environmental edge features. The network then outputs their categories, confidence levels, and bounding boxes, such as categories (such as pedestrians, cars, etc.), confidence levels (indicating the reliability of the detection results), and bounding boxes (used to determine the location of the target in the image). Semantic segmentation is then performed on the identified objects (e.g., using the DeepLab series of networks) to obtain semantic categories and achieve a structured understanding of the environment.

[0074] After the target detection network and semantic segmentation, the object to be tested is instance segmented to distinguish different semantic categories, and different instances of the same category must be distinguished, such as different pedestrians or vehicles in the image; and the object after instance segmentation is edge enhanced to extract its contour and shape, making the boundary of the object clearer, which helps to more accurately identify the shape and structure.

[0075] After a series of processing, the key target is selected from the objects to be tested in the identified target environment according to the preset category information of the key target, and the target features of the key target are extracted based on the multi-dimensional environmental data.

[0076] In this way, the present application can accurately identify key targets of preset categories through the above embodiments.

[0077] In another exemplary embodiment, the multi-dimensional environmental data further includes three-dimensional point cloud data and millimeter wave data, and the specific steps of extracting target features of key targets based on the multi-dimensional environmental data in the corresponding above embodiment may include:

[0078] Using a preset clustering algorithm based on 3D point cloud data and millimeter wave data, clusters within the target environment are detected.

[0079] Extracting features from the clusters to obtain feature data corresponding to the clusters of preset types, where the preset types include shape, position, orientation, and speed;

[0080] The data corresponding to the key target is selected from the feature data as the target feature data.

[0081] In this embodiment, the multi-dimensional environmental data includes three-dimensional point cloud data and millimeter wave data, which can be collected by lidar and millimeter wave radar. In order to identify clusters in the target environment, the RANSAC algorithm is first used to fit the ground plane to separate the ground and non-ground point clouds. The non-ground point cloud is then grouped using a preset clustering algorithm (such as the Euclidean clustering algorithm). Specifically, the non-ground point cloud is traversed. If there are enough points in the neighborhood (radius 0.3m) of a point, the cluster is expanded to output multiple clusters.

[0082] The method for extracting features from clusters can be to calculate the convex hull of each cluster, extract the minimum bounding box, and obtain the shape features and position features of the cluster; calculate the orientation angle through principal component analysis (PCA) to obtain the orientation feature; combine multi-frame tracking data to predict the motion trajectory, obtain the position feature, speed feature and orientation change, and thus obtain the feature data of the preset type.

[0083] In an exemplary embodiment provided by this application, the specific steps of obtaining weight data corresponding to each sensor in the plurality of sensors may include:

[0084] According to a preset confidence assessment mechanism, confidence assessment indicators corresponding to the multiple sensors and the confidence assessment mechanism are obtained;

[0085] Obtaining a confidence level of each sensor in the plurality of sensors based on a confidence evaluation index, and determining a first weight corresponding to each sensor based on the confidence level;

[0086] Obtain environmental status information of the target environment in real time;

[0087] Based on the environmental status information, the dynamic participation of each sensor in multi-dimensional environmental data is obtained;

[0088] determining a second weight corresponding to each sensor based on the dynamic participation degree;

[0089] The first weight is dynamically adjusted based on the second weight corresponding to each sensor to obtain weight data corresponding to each of the multiple sensors.

[0090] In this embodiment, the confidence assessment indicators may include signal quality, noise level and occlusion rate. The signal quality includes signal-to-noise ratio and signal strength; the noise level includes the noise level of the sensor data (such as radar clutter and camera image blur); the occlusion rate includes the proportion of the target being obscured (such as the camera field of view being blocked by obstacles). The confidence of the sensor is obtained based on the confidence assessment indicator and the corresponding weight information is determined, which can avoid the degree of interference of the data by unreliable sensors.

[0091] In this embodiment, environmental status information may include information such as weather, lighting, obstacle obstruction rate, etc. that may affect the quality of sensor collected data. The participation of each sensor is adjusted in real time through environmental status information. For example, the millimeter wave weight is increased at night, and the visual contribution is enhanced during the day or when the visibility conditions are good. The second weight corresponding to each sensor is determined based on the participation, and the first weight determined by the sensor confidence is adjusted based on the second weight to obtain weight data corresponding to multiple sensors.

[0092] In the embodiments provided herein, the method for adjusting the first weight determined by the sensor confidence based on the second weight may be a linear adjustment method, which is not limited herein. When the linear adjustment method is used, an adjustment coefficient is calculated for each sensor based on the second weight, and the first weight is then adjusted using this adjustment coefficient. The first weights corresponding to all the adjusted sensors are then normalized to ensure that the sum is 1, thereby obtaining weight data corresponding to each of the multiple sensors.

[0093] In this way, through the above-mentioned embodiments, the present application can evaluate the reliability of each sensor in real time, and can also automatically adjust the perception strategy according to the actual scene changes, effectively avoiding the risks brought by the failure or performance degradation of a single sensor, and significantly improving the robustness of the system under harsh conditions such as complex weather (such as rain, snow, fog), low light, occlusion and interference, so as to output more comprehensive environmental perception results.

[0094] In an exemplary embodiment provided by this application, the specific steps of determining environmental perception information of the target environment based on the three-dimensional environment map and the fused feature information may include:

[0095] Marking key targets in the three-dimensional environment map to obtain a marked map;

[0096] Based on the annotated map and fused feature information, the environmental perception information of the target environment is determined.

[0097] In this embodiment, the fused feature information includes the shape features, position features, orientation features, and speed features of the key target, so the environmental perception information can reflect the position, size (shape), and motion state of the key target in the target environment.

[0098] Figure 2 FIG. 2 is a block diagram of an environment perception device 200 using multi-sensor fusion as an exemplary embodiment of the present application. Figure 2 As shown, the device includes:

[0099] An acquisition unit 201 is configured to acquire multi-dimensional environmental data of a target environment, where the multi-dimensional environmental data is environmental data of different dimensions collected by a plurality of preset sensors;

[0100] The identification unit 202 is used to perform target identification based on the multi-dimensional environment data, and determine the key targets in the target environment and the target characteristics of the key targets;

[0101] A fusion unit 203 is configured to fuse the multi-dimensional environmental data based on weight data and target features corresponding to the multiple sensors to obtain fused data and fused feature information corresponding to the target features in the fused data, wherein the weight data includes a weight corresponding to each sensor in the multiple sensors;

[0102] A map construction unit 204 is configured to construct a three-dimensional environment map of the target environment based on the fused data;

[0103] The perception unit 205 is configured to determine environmental perception information of the target environment based on the three-dimensional environment map and the fused feature information.

[0104] The device applies the multi-sensor fusion environmental perception method provided by the present application. The recognition unit 202 identifies the key targets in the target environment through the multi-dimensional environmental data of the target environment transmitted by the multiple sensors acquired by the acquisition unit 201, and extracts the target features of the key targets based on the multi-dimensional environmental data; the fusion unit 204 fuses the multi-dimensional environmental data based on the weight data and target features corresponding to the multiple sensors to obtain the fused data and the fused feature information corresponding to the target features in the fused data; the map construction unit 205 then constructs a three-dimensional environmental map of the target environment based on the fused data; and finally, the perception unit 206 determines the environmental perception information of the target environment based on the three-dimensional environmental map and the fused feature information. In this way, the present application can process multi-source information in real time by fusing multiple sensor data, and can effectively overcome the limitations of a single sensor in a complex environment, achieving more accurate and stable environmental perception. In addition, by fusing the multi-dimensional environmental data through the weight data of each sensor, the error of a single sensor can be compensated, thereby improving the accuracy and robustness of environmental perception, reducing error and redundancy, and improving the reliability of the overall system.

[0105] In another exemplary embodiment, the target features include shape features, position features, orientation features and speed features; the fusion unit 203 is further used to use a Kalman filter fusion algorithm to fuse the dynamic data corresponding to the position features and speed features in the multidimensional environmental data based on weight data to obtain first fused feature data; use a particle filter fusion algorithm to fuse the orientation data corresponding to the orientation features in the multidimensional environmental data based on weight data to obtain second fused feature data; use a deep learning fusion algorithm to fuse the static data corresponding to the shape features in the multidimensional environmental data based on weight data to obtain third fused feature data; obtain other data in the multidimensional environmental data except the dynamic data, orientation data and static data, and fuse the other data based on weight data to obtain fourth fused data; use the first fused feature data, the second fused data and the third fused data as fused feature information corresponding to the target features; and obtain fused data of the multidimensional environmental data based on the fused feature information and the fourth fused data.

[0106] In another exemplary embodiment, the acquisition unit 201 is also used to perform consistency processing on a preset plurality of sensors to obtain a plurality of processed sensors, where the consistency processing includes acquisition time synchronization and parameter calibration; multi-dimensional environmental data of the target environment is obtained through the processed plurality of sensors, and noise filtering processing is performed on the multi-dimensional environmental data.

[0107] In another exemplary embodiment, the multi-dimensional environmental data includes image data; the recognition unit 202 is further used to perform hierarchical feature extraction on the image data based on a preset convolutional layer network to obtain environmental hierarchical features; identify the object to be tested in the target environment based on the environmental edge features through a preset target recognition network; perform semantic segmentation processing on the object to be tested to obtain the semantic category of the object to be tested; perform object division processing on the object to be tested, and identify key targets from the processed object to be tested based on the semantic category, and the object division processing includes instance segmentation and edge enhancement; and extract target features of the key targets based on the multi-dimensional environmental data.

[0108] In another exemplary embodiment, the multi-dimensional environmental data also includes three-dimensional point cloud data and millimeter wave data; the identification unit 202 is further used to detect clusters within the target environment based on the three-dimensional point cloud data and millimeter wave data through a preset clustering algorithm; perform feature extraction on the clusters to obtain feature data corresponding to preset types of clusters, the preset types including shape, position, orientation and speed; and select data corresponding to key targets from the feature data as target feature data.

[0109] In another exemplary embodiment, the apparatus further comprises:

[0110] A weighting unit is used to obtain confidence assessment indicators corresponding to multiple sensors and the confidence assessment mechanism according to a preset confidence assessment mechanism; obtain the confidence of each sensor in the multiple sensors based on the confidence assessment indicator, and determine the first weight corresponding to each sensor based on the confidence; obtain environmental status information of the target environment in real time; obtain the dynamic participation of each sensor in the multi-dimensional environmental data based on the environmental status information; determine the second weight corresponding to each sensor based on the dynamic participation; dynamically adjust the first weight based on the second weight corresponding to each sensor to obtain weight data corresponding to each of the multiple sensors.

[0111] In another exemplary embodiment, the perception unit 205 is further configured to annotate key targets in the three-dimensional environment map to obtain an annotated map; and determine environmental perception information of the target environment based on the annotated map and the fused feature information.

[0112] It should also be noted that the multi-sensor fusion environmental perception device provided by this application adopts a modular design, and data is exchanged between each functional unit through standard interfaces (such as ROStopic, services). In this way, each modular functional unit is an independent module, which is easy to maintain and upgrade. The design fully considers the needs of adding new sensors or algorithm upgrades in the future, supports inter-unit replacement and algorithm iteration, and ensures the long-term competitiveness of the technology platform. The compact structure and modular design can be flexibly expanded and customized according to specific application scenarios, and has high applicability and commercial promotion value.

[0113] It should be noted that the multi-sensor fusion environment perception device provided in the above embodiment and the multi-sensor fusion environment perception method provided in the above embodiment are of the same concept, wherein the specific manner in which each module and unit performs operations has been described in detail in the method embodiment and will not be repeated here. In actual applications, the multi-sensor fusion environment perception device provided in the above embodiment can, as needed, allocate the above functions to different functional modules, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above, and this is not limited here.

[0114] An embodiment of the present application also provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the electronic device implements the multi-sensor fusion environmental perception method provided in the above-mentioned embodiments.

[0115] Figure 3 The following is a schematic diagram showing the structure of a computer system suitable for implementing an electronic device according to an embodiment of the present application. Figure 3 The computer system 300 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0116] like Figure 3 As shown, the computer system 300 includes a central processing unit (CPU) 301, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 302 or the program loaded from the storage part 308 to the random access memory (RAM) 303, such as executing the method in the above embodiment. Various programs and data required for system operation are also stored in the RAM 303. The CPU 301, ROM 302 and RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0117] The following components are connected to the I / O interface 305: an input section 306 including a keyboard, a mouse, and the like; an output section 307 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 308 including a hard disk; and a communication section 309 including a network interface card such as a LAN (Local Area Network) card or a modem. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to the I / O interface 305 as needed. Removable media 311, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 310 as needed, so that computer programs read from the removable media can be installed in the storage section 308 as needed.

[0118] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 309, and / or installed from a removable medium 311. When the computer program is executed by the central processing unit (CPU) 301, the various functions defined in the system of the present application are executed.

[0119] It should be noted that the computer-readable medium shown in the embodiments of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, which carries a computer-readable computer program. This propagated data signal can take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. A computer program embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0120] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. Among them, each box in the flowchart or block diagram can represent a module, program segment, or part of the code, and the above-mentioned module, program segment, or part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0121] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. In some cases, the names of these units do not constitute limitations on the units themselves.

[0122] Another aspect of the present application provides a computer-readable storage medium having a computer program stored thereon. When executed by a processor, the computer program implements the aforementioned multi-sensor fusion environment perception method. The computer-readable storage medium may be included in the electronic device described in the above embodiments, or may exist independently and not be incorporated into the electronic device.

[0123] Another aspect of the present application provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the multi-sensor fusion environment perception method provided in each of the above embodiments.

[0124] The above are only preferred embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent replacements or improvements made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A multi-sensor fusion environment perception method, characterized in that: The method comprises: Acquire multi-dimensional environmental data of the target environment, wherein the multi-dimensional environmental data is environmental data of different dimensions collected by a plurality of preset sensors; Performing target recognition based on the multi-dimensional environmental data to determine key targets within the target environment and target characteristics of the key targets; Based on weight data corresponding to the multiple sensors and the target feature, the multi-dimensional environmental data is fused to obtain fused data and fused feature information corresponding to the target feature in the fused data, wherein the weight data includes a weight corresponding to each sensor in the multiple sensors; constructing a three-dimensional environment map of the target environment based on the fused data; Environmental perception information of the target environment is determined based on the three-dimensional environment map and the fused feature information.

2. The method according to claim 1, characterized in that The target features include shape features, position features, orientation features, and speed features; and the multi-dimensional environmental data is fused based on the weight data corresponding to the multiple sensors and the target features to obtain fused data and fused feature information corresponding to the target features in the fused data, including: Using a Kalman filter fusion algorithm, based on the weight data, the dynamic data corresponding to the position feature and the speed feature in the multi-dimensional environmental data are fused to obtain first fused feature data; Using a particle filter fusion algorithm, based on the weight data, the orientation data corresponding to the orientation feature in the multi-dimensional environmental data is fused to obtain second fused feature data; Using a deep learning fusion algorithm, based on the weight data, the static data corresponding to the shape feature in the multi-dimensional environmental data is fused to obtain third fused feature data; acquiring other data in the multi-dimensional environmental data except the dynamic data, the orientation data, and the static data, and fusing the other data based on the weight data to obtain fourth fused data; using the first fused feature data, the second fused data, and the third fused data as fused feature information corresponding to the target feature; The fused data of the multi-dimensional environment data is obtained based on the fused feature information and the fourth fused data.

3. The method according to claim 1, characterized in that The obtaining of multi-dimensional environmental data of the target environment includes: Performing consistency processing on the preset multiple sensors to obtain multiple processed sensors, wherein the consistency processing includes acquisition time synchronization and parameter calibration; The multi-dimensional environmental data of the target environment is acquired through the processed multiple sensors, and noise filtering is performed on the multi-dimensional environmental data.

4. The method according to claim 1, wherein The multi-dimensional environment data includes image data; the target recognition based on the multi-dimensional environment data to determine the key target in the target environment and the target characteristics of the key target includes: Performing hierarchical feature extraction on the image data based on a preset convolutional layer network to obtain environmental hierarchical features; Identifying the object to be detected in the target environment based on the edge features of the environment through a preset target recognition network; Performing semantic segmentation processing on the object to be tested to obtain a semantic category of the object to be tested; Performing object segmentation processing on the object to be tested, and identifying key targets from the processed object to be tested based on the semantic category, wherein the object segmentation processing includes instance segmentation and edge enhancement; The target features of the key target are extracted based on the multi-dimensional environmental data.

5. The method according to claim 4, characterized in that The multi-dimensional environmental data further includes three-dimensional point cloud data and millimeter wave data; and the target features of the key target extracted based on the multi-dimensional environmental data include: Detecting and obtaining clusters within the target environment based on the three-dimensional point cloud data and the millimeter wave data using a preset clustering algorithm; Performing feature extraction on the clusters to obtain feature data corresponding to the clusters of preset types, wherein the preset types include shape, position, direction, and speed; Data corresponding to the key target is selected from the feature data as target feature data.

6. The method according to claim 1 or 2, characterized in that The process of obtaining the weight data corresponding to the multiple sensors includes: According to a preset confidence assessment mechanism, obtaining confidence assessment indicators corresponding to the plurality of sensors and the confidence assessment mechanism; Obtaining a confidence level of each sensor in the plurality of sensors based on the confidence evaluation index, and determining a first weight corresponding to each sensor based on the confidence level; Acquiring environmental status information of the target environment in real time; Obtaining a dynamic participation degree of each sensor for the multi-dimensional environmental data based on the environmental state information; Determining a second weight corresponding to each sensor based on the dynamic participation degree; The first weight is dynamically adjusted based on the second weight corresponding to each sensor to obtain weight data corresponding to each of the multiple sensors.

7. The method according to claim 1, characterized in that The determining, based on the three-dimensional environment map and the fused feature information, environmental perception information of the target environment includes: Marking the key targets in the three-dimensional environment map to obtain a marked map; Based on the annotated map and the fused feature information, environmental perception information of the target environment is determined.

8. A multi-sensor fusion environment perception device, characterized in that: include: An acquisition unit, configured to acquire multi-dimensional environmental data of a target environment, wherein the multi-dimensional environmental data is environmental data of different dimensions collected by a plurality of preset sensors; an identification unit, configured to perform target identification based on the multi-dimensional environmental data, and determine key targets within the target environment, as well as target features of the key targets; a fusion unit, configured to perform a fusion process on the multi-dimensional environmental data based on weight data corresponding to the multiple sensors and the target feature, to obtain fused data and fused feature information corresponding to the target feature in the fused data, wherein the weight data includes a weight corresponding to each sensor of the multiple sensors; A map construction unit, configured to construct a three-dimensional environment map of the target environment based on the fused data; A perception unit is used to determine environmental perception information of the target environment based on the three-dimensional environment map and the fused feature information.

9. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the electronic device to implement the multi-sensor fusion environment perception method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that Computer-readable instructions are stored thereon, and when the computer-readable instructions are executed by a processor of a computer, the computer is caused to execute the multi-sensor fusion environment perception method according to any one of claims 1 to 7.

Images