An unknown obstacle detection and identification fusion perception system and method for automatic driving

Abstract

The application relates to an unknown obstacle detection and identification fusion perception system and method for automatic driving, comprising a vehicle-mounted detection unit for collecting multi-source sensor data of camera image data and laser radar point cloud data in a vehicle driving environment, and performing fusion processing on the image data and the point cloud data; a point cloud clustering unit for performing clustering analysis on residual point clouds which are not identified as known category targets by the vehicle-mounted detection unit to generate independent point cloud clusters, and screening out suspected unknown obstacles according to geometric features, motion features and time sequence continuity of the point cloud clusters. In the application, through a double-link parallel mechanism of the vehicle-mounted detection unit and the point cloud clustering unit, on one hand, a known category target is detected with high precision by using an image and point cloud fusion detection model, and on the other hand, clustering analysis is performed on residual point clouds which are not explained by the known category, so that potential unknown obstacles are found.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving, and in particular to a fusion perception system and method for detecting and recognizing unknown obstacles in autonomous driving. Background Technology

[0002] In autonomous driving technology, environmental perception is a core component for achieving safe driving. However, real-world road scenarios exhibit a significant long-tail distribution problem: approximately 99% of common data is concentrated in high-frequency categories such as vehicles and pedestrians, while long-tail obstacles such as irregularly shaped vehicles, animals, scattered goods, and temporary roadblocks, although diverse, have extremely few samples for each category, making it difficult for perception models to learn effective discriminative features. Furthermore, the obstacle category space is theoretically infinite, while the training data of existing perception systems is confined to a predefined finite set of categories, making it unable to handle the occurrence of obstacles with zero samples or unknown categories. In addition, for unknown obstacles, "object clusters" obtained solely based on point cloud clustering lack semantic descriptive capabilities; the system cannot distinguish between "cardboard boxes" and "stones," preventing the decision-making layer from accurately assessing their risk level.

[0003] To address the aforementioned issues, the following sensing solutions exist in the existing technology: (1) Schemes for detecting unknown targets based on point cloud clustering, such as using clustering algorithms like DBSCAN to cluster lidar point clouds to discover potential traffic targets. This scheme can discover entities that are outside the range of known categories, but the clustering results can only provide the geometric existence of the target and do not have semantic recognition capabilities.

[0004] (2) A scheme for abnormal target detection based on multi-sensor fusion, for example, first detects known targets through a target detection network, then generates high-confidence background proposals as candidates for unknown targets through a region proposal network, and then uses a contrastive feature clustering method to separate known and unknown categories in the feature space, and uses a multimodal large model to perform semantic recognition and category inference for unknown targets. Although this scheme initially achieves semantic exploration of unknown targets, its method of calling the multimodal large model and its computational overhead limit its real-time deployment on the vehicle.

[0005] (3) A closed-loop learning mechanism based on vehicle-cloud collaboration, for example, using multi-sensor data for local analysis at the vehicle end to initially discover abnormal road events, and then uploading suspected abnormal data to the cloud for more accurate identification and location. This solution initially constructs a data flow path from the vehicle end to the cloud, but in terms of automatic discovery of unknown obstacles, automatic assignment of semantic labels, and closed-loop update of perception models, a complete and efficient automated link has not yet been formed.

[0006] However, the aforementioned existing technical solutions still have the following shortcomings: First, there is the problem of missed detections and false positives for unidentified targets. Most existing mass-produced perception systems only detect and identify a limited number of categories defined in the training set. Objects outside the known category range are usually treated as background and ignored. When a vehicle encounters novel obstacles, the system lacks an effective warning mechanism, posing a significant safety hazard. While some open-set detection methods can uniformly label unknown targets as "unknown," they treat all unknown instances as the same, failing to distinguish between different types of unknown obstacles. Since the risk levels and response strategies for different unknown objects vary greatly, a single unknown label is insufficient to guide the vehicle in making reasonable decisions.

[0007] Second, there is the issue of semantic gaps within the fusion perception framework. While point cloud clustering methods can detect entities missed by visual detection, the clustering results lack semantic recognition capabilities. For point cloud clusters that do not match any bounding boxes, existing perception systems often treat them as noise or make a rough judgment based solely on simple volume and point count thresholds. Without the aid of image information, relying solely on point cloud geometric features is insufficient to accurately determine the true category of an object. Even if the vehicle detects the presence of an obstacle, it cannot obtain its semantic category to take appropriate countermeasures.

[0008] Third, there is a lack of a closed-loop autonomous evolution mechanism between the vehicle and the cloud. Although some industry pioneers have achieved data closure internally, few existing publicly available technical solutions involve a complete automated chain from vehicle-side online discovery of unknown targets, automatic uploading to the cloud, cloud-based semantic labeling, to continuous model updates. Currently, most autonomous driving companies rely on manually labeling long-tail data and offline training to complete model iterations, lacking real-time, autonomous evolution capabilities. There are still significant gaps in existing technologies regarding automatically collecting unknown target data, using large cloud models for automatic labeling, and efficiently updating vehicle-side models.

[0009] Fourth, there is a contradiction between large-scale model fusion deployment and real-time performance. Limited by the computing power and storage resources of onboard computing platforms, existing multimodal large-scale models for open-world object detection struggle to achieve real-time operation on the vehicle. Uploading all raw sensor data to the cloud for processing would be severely constrained by network bandwidth and transmission latency. Most existing technical solutions remain at the algorithm design level, failing to address key challenges in engineering implementation such as lightweight feature extraction and transmission, and real-time collaboration between cloud-based recognition and vehicle control decisions. Summary of the Invention

[0010] In view of this, the present invention aims to propose a fusion perception system and method for detecting and recognizing unknown obstacles for autonomous driving, in order to solve the problems in the prior art.

[0011] To achieve the above objectives, the technical solution of the present invention is implemented as follows: As a first aspect of the present invention, the present invention discloses an unknown obstacle detection and recognition fusion perception system for autonomous driving, including an on-board detection unit for collecting multi-source sensor data such as camera image data and lidar point cloud data in the vehicle driving environment, and performing fusion processing on the image data and point cloud data. The point cloud clustering unit is used to perform clustering analysis on the remaining point clouds that have not been identified as targets of known categories by the vehicle detection unit to generate independent point cloud clusters, and to screen out suspected unknown obstacles based on the geometric features, motion features and temporal continuity of the point cloud clusters. The communication unit is used to upload the feature data corresponding to the suspected unknown obstacle to the cloud and receive the identification results returned by the cloud. The cloud-based identification unit, deployed in the cloud, is used to perform multimodal fusion analysis on the feature data uploaded by the communication unit, and to perform category inference, semantic description generation, and risk level assessment on the suspected unknown obstacles through a multimodal identification model, so as to obtain the identification results of the unknown obstacles; The model update unit is used to write the new category samples and their annotation results identified and confirmed by the cloud recognition unit into the sample library, and to trigger incremental model training, performance evaluation and model update based on the new samples, so as to realize the continuous learning and optimization of the vehicle-side perception model for unknown obstacle categories.

[0012] Furthermore, the fusion process includes a fusion target detection network, which includes an image feature extraction branch for extracting multi-scale visual features, a point cloud feature extraction branch for extracting three-dimensional geometric features, and a detection head for fusing image features and point cloud features in a unified space and outputting a three-dimensional detection box and category results.

[0013] Furthermore, the clustering analysis unit includes performing preprocessing on the lidar point cloud, including ground segmentation, motion compensation, and noise removal; removing explained point clouds based on the detection boxes of known target categories output by the vehicle-mounted detection unit; and clustering the remaining unexplained point clouds using an improved clustering algorithm to obtain several independent point cloud clusters.

[0014] Furthermore, the improved clustering algorithm is a dynamic DBSCAN algorithm, whose neighborhood radius and minimum point threshold are adaptively adjusted according to point cloud distance, spatial density and radar sampling sparsity.

[0015] Furthermore, the feature data includes event identifiers, timestamps, image region ROIs, point cloud feature descriptions, visual feature vectors, target trajectory summary information, road environment context information, local risk scores, and trigger source information.

[0016] Furthermore, the cloud-based recognition unit specifically extracts visual embedding, geometric embedding, and contextual embedding from the uploaded image region, point cloud geometric features, and contextual information, respectively, and fuses the three to form a unified multimodal representation of the target to be recognized; based on the similarity calculation between the multimodal representation and the open semantic vocabulary, a candidate category set is generated; The multimodal large model is invoked to perform semantic reasoning based on the candidate category set, image region, point cloud geometric description, and contextual hints, and outputs the target category name, semantic description, judgment of whether it belongs to a new category, and risk level.

[0017] Furthermore, the model update unit includes: The sample management module is used to deduplicatize, assess the quality of, and classify and archive new category samples identified and confirmed in the cloud. The manual review module provides a manual review interface for samples identified in the cloud with a confidence level lower than a preset threshold. The incremental training module is used to incrementally train the vehicle-side model using new samples while retaining the original known category recognition performance. It adopts at least one anti-forgetting strategy among knowledge distillation, parameter regularization, or experience replay. The model evaluation and deployment module is used to perform offline evaluation and regression validation of the incrementally trained model, and to deploy the updated model to the vehicle via OTA when preset conditions are met.

[0018] As a second aspect of the present invention, the present invention discloses a method for detecting and recognizing unknown obstacles in autonomous driving, applied to the above-mentioned system, comprising the following steps: S1: Collect image data and point cloud data of the environment around the vehicle, and detect targets of a preset known category based on the image and point cloud fusion detection model; S2: Cluster the remaining point cloud that is not interpreted by the known category of target to extract suspected unknown obstacles; S3: Upload the image region, point cloud features and context information corresponding to the suspected unknown obstacle to the cloud, and the cloud will perform multimodal feature fusion and semantic reasoning to output the recognition result containing the target category, semantic description and risk level; S4: Returns the cloud recognition results to the vehicle to assist the vehicle in making control decisions such as warning, deceleration, braking or detour to unknown obstacles; S5: Store the sample data corresponding to the recognition results in the cloud sample library, and trigger incremental model training and model update distribution when preset conditions are met, so as to improve the system's ability to subsequently recognize such unknown obstacles.

[0019] Compared with the prior art, the present invention has the following advantages: In this invention, a dual-link parallel mechanism of vehicle-mounted detection unit and point cloud clustering unit is employed. On one hand, an image and point cloud fusion detection model is used to perform high-precision detection of targets of known categories. On the other hand, clustering analysis is performed on the remaining point cloud that is not interpreted by known categories to discover potential unknown obstacles. This design overcomes the limitation of traditional perception systems that can only identify closed categories, and can effectively detect long-tail obstacles outside the training set, such as irregularly shaped vehicles, scattered goods, and wild animals, significantly reducing the safety hazards caused by unknown obstacles. Attached Figure Description

[0020] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a schematic diagram of the overall architecture of the present invention; Figure 2 This is the overall flowchart of the present invention. Detailed Implementation

[0021] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0022] In the description of this invention, it should be noted that the terms "upper," "lower," "inner," and "back," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0023] Furthermore, in the description of this invention, unless otherwise explicitly defined, the terms "installation," "connection," "linking," and "connector" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention in light of the specific circumstances.

[0024] The following will refer to the appendix. Figure 1 and Figure 2 The present invention will be described in detail with reference to the embodiments.

[0025] Example 1 Overall, this embodiment discloses a fusion perception system for unknown obstacle detection and recognition in autonomous driving, which includes... The vehicle-mounted detection unit is used to collect multi-source sensor data, including camera image data and lidar point cloud data, from the vehicle's driving environment, and to perform fusion processing on the image data and point cloud data. The point cloud clustering unit is used to perform clustering analysis on the remaining point clouds that have not been identified as targets of known categories by the vehicle detection unit to generate independent point cloud clusters, and to screen out suspected unknown obstacles based on the geometric features, motion features and temporal continuity of the point cloud clusters. The communication unit is used to upload the feature data corresponding to the suspected unknown obstacle to the cloud and receive the recognition results returned by the cloud. The cloud-based identification unit, deployed in the cloud, is used to perform multimodal fusion analysis on the feature data uploaded by the communication unit, and to perform category inference, semantic description generation, and risk level assessment on suspected unknown obstacles through a multimodal identification model, so as to obtain the identification results of unknown obstacles; The model update unit is used to write the new category samples and their annotation results identified and confirmed by the cloud recognition unit into the sample library, and to trigger incremental model training, performance evaluation and model update based on the new samples, so as to realize the continuous learning and optimization of the vehicle perception model for unknown obstacle categories.

[0026] In this embodiment, the present invention employs a dual-link parallel mechanism of an onboard detection unit and a point cloud clustering unit. On one hand, it utilizes an image and point cloud fusion detection model to perform high-precision detection of targets of known categories. On the other hand, it performs clustering analysis on the remaining point cloud that is not interpreted by known categories to discover potential unknown obstacles. This design overcomes the limitation of traditional perception systems that can only identify closed categories, and can effectively detect long-tail obstacles outside the training set, such as irregularly shaped vehicles, scattered goods, and wild animals, significantly reducing the safety hazards caused by unknown obstacles.

[0027] The fusion processing includes a fusion target detection network, which includes an image feature extraction branch for extracting multi-scale visual features, a point cloud feature extraction branch for extracting three-dimensional geometric features, and a detection head for fusing image features and point cloud features in a unified space and outputting three-dimensional detection boxes and category results.

[0028] Furthermore, the clustering analysis unit includes performing preprocessing on the lidar point cloud, including ground segmentation, motion compensation, and noise removal; removing explained point clouds based on the detection boxes of known target categories output by the vehicle-mounted detection unit; and clustering the remaining unexplained point clouds using an improved clustering algorithm to obtain several independent point cloud clusters.

[0029] Based on the above settings, the improved clustering algorithm is the dynamic DBSCAN algorithm, whose neighborhood radius and minimum point threshold are adaptively adjusted according to the point cloud distance, spatial density and radar sampling sparsity.

[0030] The standard DBSCAN algorithm uses a globally uniform neighborhood radius ε, which has significant drawbacks in LiDAR point cloud scenarios: as the scanning distance increases, the point cloud density decreases significantly, and if the same ε value is still used for distant targets, the same object is easily misclassified into multiple clusters or even judged as noise. To solve the above problems, this algorithm introduces a distance-based adaptive neighborhood radius adjustment strategy. For any point pi in the remaining uninterpreted point cloud dataset, its corresponding adaptive neighborhood radius εi is determined by the following formula: in, The basic neighborhood radius can be obtained by performing nearest neighbor distance statistical analysis on the point cloud dataset. Specifically, it involves calculating the average distance between each point in the dataset and its Kth nearest neighbor. Take the corresponding value that makes the number of clusters tend to stabilize. As ri is a point The Euclidean distance to the origin of the lidar coordinate system reflects the spatial position of that point relative to the vehicle. Let be the distance adaptive coefficient function.

[0031] In one implementation, the adaptive coefficient function The Sigmoid function is used to satisfy the requirement that the neighborhood radius grows smoothly with distance: in, As the independent variable, This is a distance reference value. Based on the scaling factor, This is the dynamic gain amplitude. This is a steepness adjustment factor.

[0032] To address the aforementioned issues, this algorithm introduces a distance-based adaptive neighborhood radius adjustment strategy.

[0033] After preprocessing, based on the detection results of known target categories, point clouds that fall within the 3D detection bounding box of a known target or are associated with a known target are removed as interpreted point clouds, resulting in the remaining uninterpreted point clouds. Subsequently, the improved dynamic DBSCAN clustering algorithm is used to cluster the remaining uninterpreted point clouds, obtaining several independent point cloud clusters.

[0034] This embodiment illustrates the identification of an unknown obstacle by an autonomous vehicle equipped with the fusion perception system of this invention while driving on a real road. The unknown obstacle is a four-legged wild animal that suddenly enters the road area, specifically an adult male deer. This target does not belong to the pre-defined closed category set of the vehicle-side basic detection model (such as vehicles, pedestrians, cyclists, traffic cones, etc.), and therefore falls under the category of unknown obstacles of this invention.

[0035] This embodiment of the system includes a vehicle-side component and a cloud-side component. The vehicle-side component includes an onboard detection unit, a point cloud clustering unit, and a communication unit, deployed on the vehicle computing platform; the cloud-side component includes a cloud-based recognition unit and a model update unit, deployed on a cloud server cluster. The vehicle is equipped with at least a camera, LiDAR, a vehicle computing platform (including CPU, GPU, and memory), and a vehicle-to-everything (V2X) communication unit; the cloud-side component includes at least an access gateway, a multimodal recognition server, a sample management server, a model training server, a model evaluation server, and an OTA (Over-The-Air) release server.

[0036] In practice, during vehicle operation, the onboard detection unit collects image data from the forward-facing camera and 3D point cloud data from the LiDAR in real time. The data from each sensor are aligned with a unified timestamp via onboard middleware and spatially registered according to pre-calibrated intrinsic and extrinsic parameters.

[0037] The vehicle-mounted detection unit deploys a fusion target detection network. This network includes an image feature extraction branch and a point cloud feature extraction branch, used to extract multi-scale visual features and 3D geometric features, respectively. The two types of features are then fused in a unified space, and the detection head outputs 3D bounding boxes, category labels, and confidence scores for targets of known categories. In this embodiment, for a stag target ahead of the road, since this category is not included in the training set, the fusion target detection network failed to output a bounding box that meets the confidence threshold requirement. The point cloud corresponding to this target is marked as "unexplained point cloud" and retained.

[0038] After receiving the lidar point cloud data, the point cloud clustering unit first performs preprocessing operations, including: removing road surface point clouds using a ground plane fitting algorithm; performing motion compensation on the point clouds based on vehicle motion information to eliminate distortion; and filtering out isolated noise points and abnormal reflection points. Subsequently, based on the known target 3D detection boxes output by the vehicle-mounted detection unit, the interpreted point clouds falling within the boxes are removed. For the remaining uninterpreted point clouds, the point cloud clustering unit calls an improved dynamic DBSCAN algorithm for cluster analysis. This algorithm automatically adjusts the neighborhood radius and minimum point count threshold based on the target's distance from the radar and the point cloud density, enabling the formation of stable point cloud clusters even for sparse point clouds at medium to long distances. In this embodiment, the stag target forms an independent 3D point cloud cluster after clustering.

[0039] To reduce the probability of false targets such as vegetation disturbance and road artifacts being misidentified as obstacles, the point cloud clustering unit further filters and judges each point cloud cluster. Specifically, it extracts geometric attributes such as length, width, height, number of points, and reflection intensity statistics of the point cloud cluster, as well as dynamic attributes such as motion direction and speed obtained through multi-frame tracking, and combines these with the spatial relationship between the point cloud cluster and the lane area and shoulder area. The system uses a combination of rules and a lightweight classifier to identify only point cloud clusters that simultaneously meet the following conditions as "suspected unknown obstacles": having stable three-dimensional physical dimensions, maintaining trajectory continuity in multiple consecutive frames, and located in or near areas accessible to vehicles. In this embodiment, the stag target was successfully filtered as a suspected unknown obstacle due to its obvious volume characteristics and lateral crossing movement trend.

[0040] It should be noted that the feature data includes event identifiers, timestamps, image region ROIs, point cloud feature descriptions, visual feature vectors, target trajectory summary information, road environment context information, local risk scores, and trigger source information.

[0041] In detail, for the suspected unknown obstacles identified during the screening process, the vehicle-side system further extracts their multimodal features, including: Point cloud geometric features: the 3D bounding box size, centroid position, reflection intensity statistics, height distribution characteristics, and key point set of the target point cloud cluster; Image region features: Based on the calibration relationship between the camera and the LiDAR, point cloud clusters are projected onto the image plane to obtain the corresponding Region of Interest (ROI). Image cropping blocks are then extracted from this region, and visual feature vectors are derived. Contextual features: Environmental semantic information such as road type, time information, weather conditions, vehicle speed, and the relative motion direction of the target. The communication unit encapsulates the above feature data into an event data packet. The data packet content includes at least an event identifier, timestamp, image region ROI, point cloud feature description, visual feature vector, target trajectory summary, road environment context, and local risk score. After encryption and integrity verification, this event data packet is uploaded to the cloud access gateway via the vehicle-to-everything (V2X) channel.

[0042] It should be further explained that the cloud-based recognition unit specifically extracts visual embeddings, geometric embeddings, and contextual embeddings from the uploaded image region, point cloud geometric features, and contextual information, respectively, and merges the three to form a unified multimodal representation of the target to be identified; based on the similarity calculation between the multimodal representation and the open semantic vocabulary, a candidate category set is generated; the multimodal large model is called to perform semantic reasoning based on the candidate category set, image region, point cloud geometric description, and contextual prompts, and outputs the target category name, semantic description, judgment of whether it belongs to a new category, and risk level.

[0043] In practice, after receiving and verifying the event data packets, the cloud access gateway distributes the task to the cloud recognition unit based on the risk level assessed by the vehicle. For the stag target in this embodiment, because it is located in the potential collision zone in front of the vehicle, it is assigned a high-priority recognition task. The cloud recognition unit first performs standardization and cross-modal encoding on the uploaded data: the image ROI is input into a pre-trained visual encoder to extract visual embeddings; point cloud geometric features are input into a point cloud encoder to extract geometric embeddings; and context information is processed in a structured manner to extract context embeddings. Subsequently, the above three embeddings are fused through feature concatenation or a cross-modal attention mechanism to form a unified multimodal representation of the target. Next, the cloud recognition unit calculates the similarity between the multimodal representation and candidate categories in an open semantic vocabulary to generate an initial candidate category set (e.g., animal categories such as deer, horse, sheep, and large dogs); simultaneously, it uses point cloud geometric embeddings to constrain the candidate categories, excluding obviously inconsistent candidates based on target size, posture, and motion features, thus narrowing the candidate range.

[0044] The cloud-based recognition unit invokes a deployed multimodal large model for semantic reasoning. The large model receives image regions, point cloud geometric descriptions, and prompt templates containing road scene information. It integrates visual appearance, geometric shape, and contextual semantics to output the following results: target category name (e.g., "deer"), semantic description (e.g., "an adult male deer"), whether it belongs to a new category outside the vehicle-side known category set, and risk level. In this embodiment, the cloud-based recognition unit determines that the target is a new category, "deer," not covered by the vehicle-side model, and assigns it a high-risk level.

[0045] After generating the recognition result, the cloud-based recognition unit transmits the result back to the corresponding vehicle via the communication unit. The feedback includes at least the event identifier, target category, recognition confidence level, risk level, and suggested handling strategy. Upon receiving the feedback, the vehicle updates the target's status from "suspected unknown obstacle" to "identified obstacle" and injects the semantic tag "deer" into its local object perception list. Based on the high-risk level and execution strategy, the planning and control module triggers deceleration braking or detour maneuvers to ensure driving safety.

[0046] As a preferred embodiment, the model update unit includes: The sample management module is used to deduplicatize, assess the quality, and classify and archive new category samples identified and confirmed in the cloud. The manual review module provides a manual review interface for samples whose confidence level in the cloud identification is lower than a preset threshold. The incremental training module is used to incrementally train the vehicle-side model using new samples while retaining the original known category recognition performance. It adopts at least one anti-forgetting strategy among knowledge distillation, parameter regularization, or experience replay. The model evaluation and release module is used to perform offline evaluation and regression verification of the incrementally trained model, and release the updated model to the vehicle via OTA when preset conditions are met.

[0047] In detail, while feeding back the recognition results to the vehicle, the cloud sample management module writes the image ROI, point cloud features, context information, recognition results and confidence scores corresponding to the event into the cloud labeled sample library, and classifies and archives them according to dimensions such as category, weather, and road type.

[0048] For samples with insufficient confidence in cloud-based identification, the system transfers them to the manual review module, where remote annotators review and confirm the accuracy before writing the annotations back to the sample database.

[0049] When the cumulative number of samples for a new category (e.g., "deer") reaches a preset threshold, or when the risk level of that category is high, the model update unit triggers the incremental training process. The incremental training module mixes the newly added category samples with the existing known category samples in a proportional manner and employs a knowledge distillation strategy: using the currently deployed vehicle-side model as the teacher model, distillation constraints are applied to the new model during training. At the same time, parameter regularization techniques are used to constrain the variation of key parameters, effectively suppressing catastrophic forgetting and ensuring that the detection accuracy of the original category does not significantly decrease.

[0050] After incremental training is completed, the model evaluation module performs offline evaluation of the new model. Evaluation metrics include detection accuracy for known categories, recall rate for new categories, inference latency, and resource consumption. When all metrics of the new model meet the preset release conditions, the OTA release module updates the model, encrypts it with a signature, and wirelessly distributes it to test vehicles for gray-scale verification. After verification and stability, it is gradually pushed to all vehicles.

[0051] Therefore, when the vehicle encounters deer again during subsequent driving, the vehicle-side perception model can directly output the known category detection result of "deer" without having to go through the cloud recognition process again, thus realizing the continuous evolution of perception capabilities.

[0052] Example 2 This embodiment discloses a method for detecting and recognizing unknown obstacles in autonomous driving, which is applied to the system in Embodiment 1 above. The method includes the following steps: S1: Collect image data and point cloud data of the environment around the vehicle, and detect targets of a preset known category based on the image and point cloud fusion detection model; S2: Cluster the remaining point cloud that is not interpreted by the known category of target to extract suspected unknown obstacles; S3: Upload the image region, point cloud features and context information corresponding to the suspected unknown obstacle to the cloud, and the cloud will perform multimodal feature fusion and semantic reasoning to output the recognition result containing the target category, semantic description and risk level; S4: Returns the cloud recognition results to the vehicle to assist the vehicle in making control decisions such as warning, deceleration, braking or detour to unknown obstacles; S5: Store the sample data corresponding to the recognition results in the cloud sample library, and trigger incremental model training and model update distribution when preset conditions are met, so as to improve the system's ability to subsequently recognize such unknown obstacles.

[0053] In summary, this method can proactively detect long-tail obstacles not included in the training set, such as irregularly shaped vehicles, scattered cargo, and wild animals, significantly improving the autonomous driving system's coverage of potential safety hazards in real road environments and its response timeliness.

[0054] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A fusion perception system for detecting and recognizing unknown obstacles for autonomous driving, characterized in that: It includes an on-board detection unit for collecting multi-source sensor data, such as camera image data and lidar point cloud data, in the vehicle's driving environment, and performing fusion processing on the image data and point cloud data. The point cloud clustering unit is used to perform clustering analysis on the remaining point clouds that have not been identified as targets of known categories by the vehicle detection unit to generate independent point cloud clusters, and to screen out suspected unknown obstacles based on the geometric features, motion features and temporal continuity of the point cloud clusters. The communication unit is used to upload the feature data corresponding to the suspected unknown obstacle to the cloud and receive the identification results returned by the cloud. The cloud-based identification unit, deployed in the cloud, is used to perform multimodal fusion analysis on the feature data uploaded by the communication unit, and to perform category inference, semantic description generation, and risk level assessment on the suspected unknown obstacles through a multimodal identification model, so as to obtain the identification results of the unknown obstacles; The model update unit is used to write the new category samples and their annotation results identified and confirmed by the cloud recognition unit into the sample library, and to trigger incremental model training, performance evaluation and model update based on the new samples, so as to realize the continuous learning and optimization of the vehicle-side perception model for unknown obstacle categories.

2. The unknown obstacle detection and recognition fusion perception system for autonomous driving according to claim 1, characterized in that: The fusion process includes a fusion target detection network, which includes an image feature extraction branch for extracting multi-scale visual features, a point cloud feature extraction branch for extracting three-dimensional geometric features, and a detection head for fusing image features and point cloud features in a unified space and outputting a three-dimensional detection box and category results.

3. The unknown obstacle detection and recognition fusion perception system for autonomous driving according to claim 1, characterized in that: The clustering analysis unit includes performing preprocessing on the lidar point cloud, including ground segmentation, motion compensation, and noise removal; removing explained point clouds based on the detection boxes of known target categories output by the vehicle-mounted detection unit; and clustering the remaining unexplained point clouds using an improved clustering algorithm to obtain several independent point cloud clusters.

4. The unknown obstacle detection and recognition fusion perception system for autonomous driving according to claim 3, characterized in that: The improved clustering algorithm is the dynamic DBSCAN algorithm, whose neighborhood radius and minimum point threshold are adaptively adjusted according to point cloud distance, spatial density and radar sampling sparsity.

5. The unknown obstacle detection and recognition fusion perception system for autonomous driving according to claim 1, characterized in that: The feature data includes event identifiers, timestamps, image region ROIs, point cloud feature descriptions, visual feature vectors, target trajectory summary information, road environment context information, local risk scores, and trigger source information.

6. The unknown obstacle detection and recognition fusion perception system for autonomous driving according to claim 1, characterized in that: The cloud-based recognition unit specifically extracts visual embedding, geometric embedding, and contextual embedding from the uploaded image region, point cloud geometric features, and contextual information, and then fuses the three to form a unified multimodal representation of the target to be recognized. Based on the similarity calculation between the multimodal representation and the open semantic vocabulary, a candidate category set is generated; The multimodal large model is invoked to perform semantic reasoning based on the candidate category set, image region, point cloud geometric description, and contextual hints, and outputs the target category name, semantic description, judgment of whether it belongs to a new category, and risk level.

7. The unknown obstacle detection and recognition fusion perception system for autonomous driving according to claim 1, characterized in that: The model update unit includes: The sample management module is used to deduplicatize, assess the quality of, and classify and archive new category samples identified and confirmed in the cloud. The manual review module provides a manual review interface for samples identified in the cloud with a confidence level lower than a preset threshold. The incremental training module is used to incrementally train the vehicle-side model using new samples while retaining the original known category recognition performance. It adopts at least one anti-forgetting strategy among knowledge distillation, parameter regularization, or experience replay. The model evaluation and deployment module is used to perform offline evaluation and regression validation of the incrementally trained model, and to deploy the updated model to the vehicle via OTA when preset conditions are met.

8. A method for detecting and recognizing unknown obstacles in autonomous driving, applied to the system described in any one of claims 1 to 7, characterized in that, Includes the following steps: S1: Collect image data and point cloud data of the environment around the vehicle, and detect targets of a preset known category based on the image and point cloud fusion detection model; S2: Cluster the remaining point cloud that is not interpreted by the known category of target to extract suspected unknown obstacles; S3: Upload the image region, point cloud features and context information corresponding to the suspected unknown obstacle to the cloud, and the cloud will perform multimodal feature fusion and semantic reasoning to output the recognition result containing the target category, semantic description and risk level; S4: Returns the cloud recognition results to the vehicle to assist the vehicle in making control decisions such as warning, deceleration, braking or detour to unknown obstacles; S5: Store the sample data corresponding to the recognition results in the cloud sample library, and trigger incremental model training and model update distribution when preset conditions are met, so as to improve the system's ability to subsequently recognize such unknown obstacles.

Images