Passable area determination method and device, vehicle and electronic equipment thereof
By using a multi-sensor combination method to determine recognition confidence and fusion confidence, the problem of inaccurate identification of passable areas by a single sensor in a strong interference environment is solved, enabling accurate identification of passable areas and safe driving of vehicles in complex environments.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHERY COMMERCIAL VEHICLE (SHANDONG) TECHNOLOGY CO LTD
- Filing Date
- 2026-01-14
- Publication Date
- 2026-05-19
AI Technical Summary
In the field of intelligent driving, a single sensor is difficult to accurately identify passable areas of the road in a strong interference environment, resulting in large errors in the identification results and affecting the reliability of the vehicle's intelligent driving function.
By combining multiple sensors (image sensor, lidar sensor and millimeter-wave radar sensor), and by determining the identification confidence and fusion confidence, real-time data from each sensor is fused to accurately identify the passable area of the vehicle in a strong interference environment.
It achieves accuracy and safety in identifying passable areas of vehicles under strong interference, avoids errors caused by a single sensor, and improves the reliability of intelligent driving functions.
Smart Images
Figure CN122067413A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of intelligent driving, and more specifically, to a method, apparatus, vehicle, and electronic equipment for determining a passable area. Background Technology
[0002] In the field of intelligent driving, the ability of a vehicle to accurately identify passable areas on the road is one of the crucial foundations for realizing intelligent driving functions. Currently, identifying passable areas on the road is typically done using a single sensor, which already demonstrates high reliability under normal weather and lighting conditions. However, in environments with strong interference, the perception gaps caused by a single sensor become apparent, and environmental factors lead to large errors in the identification results, exhibiting significant defects. This, in turn, limits the sensor's functionality, causing the vehicle's intelligent driving functions to disengage. Therefore, how to accurately determine passable areas under strong interference environments has become an urgent problem to be solved. Summary of the Invention
[0003] In view of this, embodiments of this application propose a method, apparatus, vehicle, and electronic equipment for determining passable areas to improve the above-mentioned problems.
[0004] According to a first aspect of the embodiments of this application, a method for determining a passable area is provided. The method includes: acquiring real-time data collected by each of a plurality of sensors of a vehicle, wherein the plurality of sensors includes any combination of an image sensor, a lidar sensor, and a millimeter-wave radar sensor; determining the recognition confidence level of each of the plurality of sensors based on the real-time data; determining the fusion confidence level corresponding to the real-time data of each sensor based on the data type corresponding to the real-time data; and determining the passable area of the vehicle in the current environment based on the recognition confidence level of each sensor, the fusion confidence level corresponding to the real-time data of each sensor, and the real-time data.
[0005] In some embodiments, determining the recognition confidence level of each of the plurality of sensors based on the real-time data includes: matching the real-time data collected by each sensor with each sample data in the sample database to determine the matching degree corresponding to the real-time data of each sensor; and determining the recognition confidence level of each of the plurality of sensors based on the matching degree.
[0006] In some embodiments, determining the fusion confidence level of the real-time data of each sensor in the real-time data based on the data type corresponding to the real-time data includes: determining the data features of the real-time data of each sensor corresponding to the data type, wherein the data type includes real-time image data, real-time first point cloud data, and real-time second point cloud data, and the data features include image signal-to-noise ratio, point cloud signal integrity, and spatial coverage; determining the fusion confidence level corresponding to the real-time data of each sensor according to the data features, wherein the fusion confidence level characterizes the proportion of real-time data indicated by the data features that is fused.
[0007] In some embodiments, determining the fusion confidence level corresponding to the real-time data of each sensor based on the data features includes: determining a first fusion confidence level of the real-time image data based on the image signal-to-noise ratio; determining a second fusion confidence level of the first point cloud real-time data based on the point cloud signal integrity; and determining a third fusion confidence level of the second point cloud real-time data based on the spatial coverage.
[0008] In some embodiments, the sum of the confidence levels of the first fusion confidence level, the second fusion confidence level, and the third fusion confidence level is equal to a confidence threshold.
[0009] In some embodiments, determining the passable area of the vehicle in the current environment based on the identification confidence level of each sensor, the fusion confidence level corresponding to the real-time data of each sensor, and the real-time data of each sensor includes: determining multiple reference passable areas based on the real-time data of each sensor; and fusing the multiple reference passable areas based on the identification confidence level and the fusion confidence level to obtain the passable area.
[0010] According to a second aspect of the embodiments of this application, a device for determining a passable area is provided. The device includes: an acquisition module, configured to acquire real-time data collected by each of a plurality of sensors of a vehicle, wherein the plurality of sensors includes any combination of an image sensor, a lidar sensor, and a millimeter-wave radar sensor; an identification confidence determination module, configured to determine the identification confidence of each of the plurality of sensors based on the real-time data; a fusion confidence determination module, configured to determine the fusion confidence corresponding to the real-time data of each sensor based on the data type corresponding to the real-time data; and a passable area determination module, configured to determine the passable area of the vehicle in the current environment based on the identification confidence of each sensor, the fusion confidence corresponding to the real-time data of each sensor, and the real-time data.
[0011] According to a third aspect of the embodiments of this application, a vehicle is provided, comprising: a plurality of sensors, wherein the plurality of sensors include any combination of an image sensor, a lidar sensor, and a millimeter-wave radar sensor; and a processing module, the processing module being configured to receive data collected by the plurality of sensors and implement the method for determining the passable area as described above.
[0012] According to a fourth aspect of the embodiments of this application, an electronic device is provided, comprising: a processor; and a memory storing computer-readable instructions, wherein when the computer-readable instructions are executed by the processor, the method for determining a passable area as described above is implemented.
[0013] According to a fifth aspect of the embodiments of this application, a computer-readable storage medium is provided, on which computer-readable instructions are stored, which, when executed by a processor, implement the method for determining a passable area as described above.
[0014] In this application, the identification confidence level of each sensor is first determined based on the real-time data collected by each of the vehicle's multiple sensors. Then, the fusion confidence level of the real-time data is determined based on the data type of the real-time data collected by each sensor. Finally, the passable area of the vehicle in the current environment is determined based on the identification confidence level of each sensor, the fusion confidence level of the real-time data of each sensor, and the real-time data. This approach uses multiple different types of sensors to jointly determine the passable area of the vehicle in the environment, avoiding the problem of large errors in determining the passable area using a single sensor. This application achieves passable area identification in complex multi-sensor scenarios by performing perception fusion based on the identification confidence level of each sensor and the fusion confidence level of the real-time data of each sensor, and outputting the safe passable area in real time. This ensures accurate determination of the passable area even in environments with strong interference.
[0015] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit the embodiments of this application. Attached Figure Description
[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.
[0017] Figure 1This is a schematic diagram of a vehicle according to an embodiment of this application.
[0018] Figure 2 This is a flowchart illustrating a method for determining a passable area according to an embodiment of this application.
[0019] Figure 3 This is a flowchart illustrating a method for determining a passable area according to another embodiment of this application.
[0020] Figure 4 This is a flowchart illustrating a method for determining a passable area according to another embodiment of this application.
[0021] Figure 5 This is a schematic flowchart illustrating a method for determining a passable area according to another embodiment of this application.
[0022] Figure 6 This is a schematic flowchart illustrating a method for determining a passable area according to another embodiment of this application.
[0023] Figure 7 This is a block diagram of a device for determining a passable area according to an embodiment of this application.
[0024] Figure 8 This is a hardware structure diagram of an electronic device according to an embodiment of this application.
[0025] The accompanying drawings have illustrated specific embodiments of the present application. More detailed descriptions will follow. These drawings and descriptions are not intended to limit the scope of the present application's embodiments in any way, but rather to illustrate the concepts of the present application's embodiments to those skilled in the art through specific embodiments. Detailed Implementation
[0026] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this application more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art.
[0027] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.
[0028] Please see Figure 1 , Figure 1 A vehicle provided in one embodiment of this application is shown, such as Figure 1 As shown below, the method for determining the passage area for vehicles will be illustrated by example.
[0029] In one alternative implementation, the vehicle 100 includes a camera module 110, a millimeter-wave radar module 120, a lidar module 130, and a processing module 140, wherein the camera module 110, the millimeter-wave radar module 120, and the lidar module 130 refer to hardware devices, and the processing module 140 refers to a software unit or module.
[0030] For example, the camera module 110 is used to perform image perception on the vehicle's environment to obtain image information; the millimeter-wave radar module 120 is used to emit millimeter waves and receive the echo information of the returned millimeter waves to perceive the millimeter-wave echo information of moving obstacles around the vehicle; the lidar module 130 is used to perceive the point cloud data of other objects in the vehicle's environment to obtain point cloud information; the processing module 140 is used to perform perception and recognition based on the image information collected by the camera module 110, the millimeter-wave radar module 120, and the lidar module 130 to determine the passage areas corresponding to the camera module 110, the millimeter-wave radar module 120, and the lidar module 130, so that the passable area can be comprehensively determined based on the passable areas corresponding to the camera module 110, the millimeter-wave radar module 120, and the lidar module 130.
[0031] For example, the camera module 110 may include an 8MP telephoto camera with a 30° field of view and an 8MP wide-angle camera with a 120° field of view, thereby achieving visual perception that combines both near and far perspectives and coarse and fine details. The millimeter-wave radar module 120 may include a front radar and corner radars (front left corner radar and front right corner radar) located in front of the vehicle, used to accurately measure the speed, distance, etc. of all participants in the vehicle's environment, and to improve detection accuracy by covering the blind spots on the sides and front through the corner radar. The lidar module 130 may include a semi-solid-state lidar, thereby enabling the detection of dense and precise three-dimensional point clouds, and directly measuring the shape, size, and ground contour of objects.
[0032] Figure 1 The vehicles in the document can be used to achieve the following Figure 2 For the method used to determine the described passable area, please refer to [link / reference]. Figure 2 , Figure 2 This application illustrates a method for determining a passable area according to an embodiment of the present application. In a specific embodiment, this method for determining a passable area can be applied to, for example... Figure 7The device 600 for determining the passable area and the vehicle 100 equipped with the device 600 for determining the passable area are shown. Figure 1 ) or electronic device 700 ( Figure 8 The specific process of this embodiment will be described below. Of course, it is understood that this method can be executed by an electronic device with computing power, such as a vehicle-mounted server, a cloud server, or other processors. The following will focus on... Figure 2 The process shown is described in detail. The method for determining the passable area may specifically include the following steps 210-240.
[0033] Step 210: Obtain real-time data collected by each of the vehicle's multiple sensors, wherein the multiple sensors include any combination of image sensors, lidar sensors, and millimeter-wave radar sensors.
[0034] As an alternative approach, in order to accurately identify passable areas, multiple sensors of various types can be pre-installed at different locations on the vehicle. This allows multiple sensors to collect real-time data on the vehicle during its journey, and the passable areas can then be identified using the real-time data collected by each sensor.
[0035] Optionally, to further ensure the accuracy of the determined passable area for the vehicle, multimodal real-time data of the vehicle during its operation can be collected. This data can then be identified and fused to comprehensively determine the passable area. This reduces the error caused by using a single sensor to determine the passable area and improves the safety of the vehicle's intelligent driving functions. Specifically, an image sensor acquires image information from the vehicle's vision system; a millimeter-wave radar sensor acquires target and raw point cloud information perceived by millimeter-wave echoes; and a lidar sensor acquires lidar-scanned road surface echo signals.
[0036] Optionally, since the multiple sensors are of different types, the real-time data acquired by each sensor is also different. For example, the real-time data acquired by the image sensor is real-time image data, the real-time data acquired by the LiDAR sensor is real-time 3D point cloud data, and the real-time data acquired by the millimeter-wave radar sensor is real-time target point trace / list data or real-time sparse point cloud data.
[0037] Step 220: Determine the recognition confidence level of each of the plurality of sensors based on the real-time data.
[0038] As an alternative approach, after acquiring real-time data from each of the vehicle's multiple sensors, the recognition confidence level of the corresponding sensor can be determined based on the real-time data collected by each sensor. This allows for real-time adjustments to the impact of determining passable areas based on the real-time data of each sensor, ensuring the accuracy of passable areas.
[0039] In one alternative scenario, a scene library can be pre-set in the vehicle's local database or cloud database to convert scenes in different environments into corresponding scene data information. After acquiring the real-time data of each sensor, the real-time data of each sensor is compared with the scene data information in the scene library to determine the target scene that is closest to the real-time data of each sensor in the scene library. Based on the target scene, the recognition confidence of each sensor among multiple sensors is determined.
[0040] Step 230: Based on the data type corresponding to the real-time data, determine the fusion confidence level of the real-time data of each sensor in the real-time data.
[0041] As an alternative approach, after acquiring the real-time data from each sensor, the data type of the real-time data can be determined first. Then, based on the data type of the real-time data from each sensor, the fusion confidence level of the real-time data from each sensor can be determined. Finally, based on the fusion confidence level of the real-time data from each sensor, the passable area of the vehicle during driving can be comprehensively determined, thereby ensuring the accuracy of the determined passable area.
[0042] In one alternative scenario, since real-time data of different data types has its unique characteristics, the impact of these characteristics on the size of the passable area varies depending on the data type used to determine the passable area. Therefore, the data types corresponding to multiple sets of real-time data can be determined first. Then, the fusion confidence level of the real-time data can be determined based on the data characteristics corresponding to each data type. Finally, the passable area can be determined based on the fusion confidence level. Optionally, the data types may include radar point cloud data, image data, and sparse point cloud data, with the specific data type depending on the corresponding sensor.
[0043] Step 240: Determine the passable area of the vehicle in the current environment based on the identification confidence of each sensor, the fusion confidence of the real-time data of each sensor, and the real-time data.
[0044] As an alternative approach, after determining the fusion confidence level and the identification confidence level of each sensor's real-time data, in order to determine the vehicle's passable area in the current environment, the real-time data can first be perceptually fused based on the identification confidence level and the fusion confidence level of each sensor's real-time data to obtain fused data. Then, the fused data can be identified to determine the vehicle's passable area in the current environment.
[0045] In one alternative scenario, the determination of a passable area in a vehicle's environment by a single sensor or a single type of sensor may be inaccurate due to the limitations of the sensor itself. Therefore, to improve the accuracy of the determined passable area, real-time data from different types of sensors can be fused together to determine the passable area based on the fused data.
[0046] In one optional scenario, a comprehensive weight can be determined first based on the recognition confidence of each sensor and the fusion confidence of the real-time data from each sensor. Simultaneously, real-time data from multiple sensors are recognized to determine the passability probability of the vehicle's environment. Then, the passable area within the vehicle's environment is determined based on the comprehensive weight and the passability probability. Optionally, if the recognition confidence and fusion confidence of a certain sensor contradict each other in a specific scenario, a comprehensive weight can be set to significantly suppress its value in order to ensure the accuracy of the passable area determination. This avoids the increased error in the determined passable area caused by the contradiction between the confidence and the weight.
[0047] Optionally, in order to accurately determine passable areas, real-time data collected by multiple sensors can be projected onto a grid map to determine the probability that each grid in the grid map is in an idle state, thereby determining the passability probability of each grid in the vehicle's environment.
[0048] In the embodiments of this application, the recognition confidence level of each sensor is first determined based on the real-time data collected by each of the multiple sensors of the vehicle. Then, the fusion confidence level of the real-time data of each sensor is determined based on the data type of the real-time data collected by each sensor. Finally, the passable area of the vehicle in the current environment can be determined based on the recognition confidence level of each sensor, the fusion confidence level of the real-time data of each sensor, and the real-time data. This approach uses multiple different types of sensors to jointly determine the passable area of the vehicle in the environment, avoiding the problem of large errors in determining the passable area using a single sensor. This application achieves passable area recognition in complex multi-sensor scenarios by performing perception fusion based on the recognition confidence level of each sensor and the fusion confidence level of the real-time data of each sensor, and outputting the safe passable area in real time. This ensures accurate determination of the passable area even when the vehicle is in a strong interference environment.
[0049] Please see Figure 3 , Figure 3 This application illustrates a method for determining a passable area according to an embodiment of the present application. The following will focus on... Figure 3 The process shown is described in detail, and the method for determining the passable area may specifically include the following steps 310-350.
[0050] Step 310: Acquire real-time data collected by each of the vehicle's multiple sensors, wherein the multiple sensors include any combination of image sensors, lidar sensors, and millimeter-wave radar sensors.
[0051] Step 320: Match the real-time data collected by each sensor with each sample data in the sample database to determine the matching degree corresponding to the real-time data of each sensor.
[0052] As an alternative approach, in order to accurately determine the recognition confidence level of each sensor, a sample database can be pre-set. This sample database includes sample data corresponding to multiple different sensors under different interference environments. This allows the matching degree of each sensor's real-time data to be determined by matching the real-time data collected by each sensor with each sample data in the sample database.
[0053] In one alternative scenario, under different interference environments, the real-time data detected by different types of sensors have varying degrees of accuracy in determining passable areas. Therefore, the real-time data can be matched with each sample data in the sample data path to obtain the matching degree corresponding to the real-time data of each sensor. Then, based on the matching degree, the interference environment corresponding to multiple sensors can be determined in the matched sample data, thereby determining the recognition confidence of each sensor.
[0054] Step 330: Determine the recognition confidence level of each of the plurality of sensors based on the matching degree.
[0055] As an alternative approach, after determining the matching degree corresponding to the real-time data of each sensor, the target sample data group with the highest matching degree is determined in the sample database. Then, based on the target sample data group, the target interference environment corresponding to the real-time data of multiple sensors is determined. In this way, the recognition confidence of each sensor among multiple sensors can be determined based on the target interference environment.
[0056] In one alternative scenario, to accurately determine the recognition confidence level of each sensor, sample data sets under various interference environments can be pre-set in the sample database. For each interference environment, a different correspondence can be established between the recognition confidence level of each sensor and the interference environment. Thus, after determining the matching degree corresponding to the real-time data of each sensor, the target sample data set corresponding to the real-time data can be determined based on this matching degree. Furthermore, the recognition confidence level of each sensor can be determined based on this correspondence and the target sample data set.
[0057] For example, in heavy rain conditions, the confidence level of a lidar sensor decreases, while the confidence level of millimeter-wave radar and image sensors increases. This ensures that the real-time data collected by millimeter-wave radar and image sensors is more reliable in heavy rain. However, due to the heavy rain, the real-time data collected by the lidar sensor differs significantly from the actual road conditions, resulting in substantial errors. Therefore, the confidence level of the lidar sensor can be appropriately reduced to ensure the accuracy of the determined passable area.
[0058] Step 340: Based on the data type corresponding to the real-time data, determine the fusion confidence level of the real-time data of each sensor in the real-time data.
[0059] Step 350: Determine the passable area of the vehicle in the current environment based on the identification confidence of each sensor, the fusion confidence of the real-time data of each sensor, and the real-time data.
[0060] The specific steps of steps 310 and 340-350 can be found in steps 210 and 230-240, and will not be repeated here.
[0061] In this embodiment, the matching degree of each sensor's real-time data is determined by matching the real-time data collected by each sensor with each sample data in the sample database. This allows the identification confidence of each sensor among multiple sensors to be determined based on the matching degree, ensuring the accuracy of the identification confidence of each sensor and further guaranteeing the accuracy of the determined passable area.
[0062] Please see Figure 4 , Figure 4 This application illustrates a method for determining a passable area according to an embodiment of the present application. The following will focus on... Figure 4 The process shown is described in detail. The method for determining the passable area may specifically include the following steps 410-450.
[0063] Step 410: Obtain real-time data collected by each of the vehicle's multiple sensors, wherein the multiple sensors include any combination of image sensors, lidar sensors, and millimeter-wave radar sensors.
[0064] Step 420: Determine the recognition confidence level of each of the plurality of sensors based on the real-time data.
[0065] Step 430: Determine the data characteristics of the real-time data of each sensor corresponding to the data type, wherein the data type includes real-time image data, real-time first point cloud data, and real-time second point cloud data, and the data characteristics include image signal-to-noise ratio, point cloud signal integrity, and spatial coverage.
[0066] As an alternative approach, since real-time data of different data types correspond to different data characteristics, we can first determine multiple reference features corresponding to the real-time data type of each sensor based on the data type of the real-time data of each sensor, and then determine the data features corresponding to the real-time data from among the multiple reference features. In this way, we can determine the recognition confidence of each sensor based on the data features.
[0067] In one alternative scenario, since the quality of real-time image data directly affects the recognition result during the recognition process, the image signal-to-noise ratio (SNR), as a data feature, can intuitively reflect the quality of real-time image data. The SNR of real-time image data can be determined using methods such as the uniform region method, frame difference method, wavelet transform method, and local variance method.
[0068] Optionally, during the identification process of the first point cloud real-time data, environmental factors may cause the presence of some false point clouds in the data, leading to a decrease in the accuracy of obstacle identification and further errors in determining the passable area. Therefore, the integrity of the point cloud signal in the first point cloud real-time data can be determined, thereby indicating the accuracy of obstacle detection in the surrounding environment. Specifically, the integrity of the point cloud signal can be determined by assessing the number of abnormal noise points in the first point cloud real-time data and the consistency of multiple returns from the multi-return radar.
[0069] Optionally, since there may be some perception blind spots during the identification of the second point cloud real-time data, to avoid errors in determining the passable area due to inaccurate identification of the second point cloud real-time data caused by perception blind spots, the spatial coverage rate of the second point cloud real-time data can be determined. This spatial coverage rate can then be used to determine the degree of coverage of the surrounding environment. Specifically, the sensor's detection space can be discretized into multiple small volume units or angle-distance grids. Then, each point in the second point cloud real-time data is traversed, transformed into a polar coordinate system (or spherical coordinate system) with the sensor as the origin, and then mapped to the corresponding grid. Next, the number of occupied grids is counted, and finally, this number is divided by the total number of grids to obtain the spatial coverage rate.
[0070] Step 440: Determine the fusion confidence level corresponding to the real-time data of each sensor based on the data features, wherein the fusion confidence level represents the proportion of real-time data indicated by the data features that are fused.
[0071] As an alternative approach, since the accuracy of real-time data collected by different sensors varies in different environments, in order to ensure the accuracy of the passable area determined based on the real-time data, the fusion confidence level corresponding to the real-time data of each sensor can be determined based on the data characteristics of the data type corresponding to the real-time data of each sensor. In this way, the passable area can be determined based on the fusion confidence level and the corresponding real-time data.
[0072] In some embodiments, step 440 includes: determining a first fusion confidence level of the real-time image data based on the image signal-to-noise ratio; determining a second fusion confidence level of the first real-time point cloud data based on the point cloud signal integrity; and determining a third fusion confidence level of the second real-time point cloud data based on the spatial coverage.
[0073] As an alternative approach, to ensure the accuracy of the passable area, the image signal-to-noise ratio (SNR) of the real-time image data, the point cloud signal integrity of the first point cloud real-time data, and the spatial coverage of the second point cloud real-time data can be determined separately. The image SNR of the real-time image data can be calculated using the formula... To determine the first reference confidence level, where, The first reference confidence level is given, and SNR is the image signal-to-noise ratio of the real-time image data. This is the visual weight adjustment coefficient, which can be set according to actual needs. Then, the first reference confidence score is normalized using the Sigmoid function to obtain the first fusion confidence score, which can be obtained using the formula... To normalize the first reference confidence level, where, The first fusion confidence level, This is the first adjustment coefficient, used to control the slope of the fusion confidence distribution.
[0074] Optionally, regarding the point cloud signal integrity of the first point cloud real-time data, the number of effective millimeter-wave point clouds and the theoretical maximum number of point clouds in the first point cloud real-time data can be determined first. Based on these, a second reference confidence level can be determined, which can be calculated using the formula... To determine the second reference confidence level, where, As the second reference confidence level, To determine the effective number of millimeter-wave point clouds, This represents the theoretical maximum number of point clouds. Then, the radar signal quality score (e.g., signal-to-noise ratio, target stability) and the quality score upper limit are determined. Based on this, the second fusion confidence level of the first point cloud real-time data is determined using the radar signal quality score, the quality score upper limit, and the second reference confidence level. This can be achieved through the formula... To determine the second fusion confidence, where... For the second fusion confidence level, Scoring the radar signal quality This represents the upper limit of the quality score.
[0075] Optionally, regarding the spatial coverage of the second point cloud real-time data, the third reference confidence level can be determined first based on the spatial volume of the effective point cloud coverage and the scan volume of the second point cloud real-time data, using the formula... To determine the third reference confidence level, As the third reference confidence level, For the effective cloud coverage space volume, This refers to the scan volume of the second point cloud real-time data. The spatial coverage of the second point cloud real-time data is used. Then, the point cloud density of the second point cloud real-time data can be further combined to optimize the third reference confidence score, resulting in the third fusion confidence score, which can be obtained through the formula... To determine the third fusion confidence, where, For the third fusion confidence level, The point cloud density of the current frame. This represents the nominal point cloud density.
[0076] In some embodiments, the sum of the confidence levels of the first fusion confidence level, the second fusion confidence level, and the third fusion confidence level is equal to a confidence threshold.
[0077] As an alternative approach, in order to better combine any combination of image sensors, lidar sensors, and millimeter-wave radar sensors to comprehensively determine the passable area of a vehicle in the current environment, and to ensure consistency, rationality, and stability during the fusion of real-time data from each sensor, the sum of the confidence levels of the first fusion confidence level, the second fusion confidence level, and the third fusion confidence level can be limited to equal a confidence threshold.
[0078] Step 450: Determine the passable area of the vehicle in the current environment based on the identification confidence of each sensor, the fusion confidence of the real-time data of each sensor, and the real-time data.
[0079] The specific steps of steps 410-420 and 450 can be found in steps 210-220 and 240, and will not be repeated here.
[0080] In this embodiment, by determining the data characteristics corresponding to the data type of the real-time data of each sensor, the fusion confidence of the proportion of the real-time data of each sensor to the real-time data indicated by the data characteristics can be determined based on the data characteristics, thus ensuring the accuracy of the determined fusion confidence and further ensuring the accuracy of the determined passable area.
[0081] Please see Figure 5 , Figure 5 This application illustrates a method for determining a passable area according to an embodiment of the present application. The following will focus on... Figure 5 The process shown is described in detail, and the method for determining the passable area may specifically include the following steps 510-550.
[0082] Step 510: Obtain real-time data collected by each of the vehicle's multiple sensors, wherein the multiple sensors include any combination of image sensors, lidar sensors, and millimeter-wave radar sensors.
[0083] Step 520: Determine the recognition confidence level of each of the plurality of sensors based on the real-time data.
[0084] Step 530: Based on the data type corresponding to the real-time data, determine the fusion confidence level of the real-time data of each sensor in the real-time data.
[0085] The specific steps of steps 510-530 can be found in steps 210-230, and will not be repeated here.
[0086] Step 540: Determine multiple reference passage areas based on the real-time data from each sensor.
[0087] As an alternative approach, the reference passage area corresponding to the real-time data collected by each sensor can be determined first based on the real-time data of each sensor, thereby obtaining multiple reference passage areas. The passable area can then be determined by fusing these multiple reference passage areas.
[0088] Optionally, for real-time image data, a neural network can be used to identify the real-time image data and obtain identification results. The first reference passage area is then determined based on the empty roads indicated in the identification results. For the second real-time point cloud data collected by a lidar sensor, an obstacle detection algorithm can be used to determine obstacles within the detection range of the second point cloud data and obtain detection results. The second reference passage area is then determined based on these detection results. For the first real-time point cloud data collected by a millimeter-wave radar sensor, Doppler velocity information is used to separate target point clouds with relative velocity. Then, clustering is used to aggregate the target point clouds to identify obstacles. The obstacle area is then projected onto a two-dimensional grid map, and each grid cell in the two-dimensional grid map is marked to obtain an occupied grid map, which includes markings of whether each grid cell is occupied. Finally, a third reference passage area is determined based on the occupied grid map.
[0089] Step 550: The multiple reference passage areas are fused according to the identification confidence and the fusion confidence to obtain the passable area.
[0090] As an alternative approach, in order to determine the passable area within the residency period, the target weight corresponding to each reference passable area can be determined first based on the identification confidence and fusion confidence. Then, multiple reference passable areas can be fused based on the target weight to obtain the passable area.
[0091] In one alternative scenario, when fusing multiple reference passage areas based on the target weight, the overlapping and non-overlapping parts of the multiple reference passage areas can be determined first. Then, the non-overlapping parts can be fused according to the target weight to obtain a sub-passage area. Finally, the passable area can be obtained based on the sub-passage area and the overlapping part.
[0092] Optionally, for multiple non-overlapping parts, the state of each grid cell in the corresponding two-dimensional grid can be determined first, wherein each grid cell state includes an initial probability of being in an occupied state. Then, based on the target weight and the initial probability of each grid cell being in an occupied state, the target probability of each grid cell being in an occupied state is determined. Then, the grid cells with target probabilities less than the probability threshold are determined as sub-passable areas, thereby enabling the obtainable area based on the sub-passable areas and overlapping parts.
[0093] In this embodiment, multiple reference passage areas can be determined in advance based on the real-time data of each sensor. Then, the identification confidence and fusion confidence are combined to fuse the multiple reference passage areas to obtain the passable area, thus ensuring the accuracy of the passable area.
[0094] Figure 6 This is a method for determining a passable area according to an embodiment of this application, such as... Figure 6 As shown, firstly, high-precision maps and high-precision map information are acquired. The high-precision map information may include obstacle information, road surface type, or weather type, etc. Real-time data of various data types is acquired using camera sensors, millimeter-wave radar sensors, and lidar sensors. Then, environmental interference is assessed based on the real-time data of various data types and the high-precision map information to determine the recognition confidence levels of each of the camera sensors, millimeter-wave radar sensors, and lidar sensors, as well as the fusion confidence levels of the real-time data of various data types. Next, perception fusion is performed based on the recognition confidence levels of the camera sensors, millimeter-wave radar sensors, and lidar sensors, the fusion confidence levels of the real-time data of various data types, and the acquired traffic information to determine the passable area. Finally, the vehicle's intelligent driving system controls the vehicle based on the determined passable area.
[0095] The above embodiments describe in detail the method for determining a passage area provided in the embodiments of this application. In other embodiments, this application also provides a device for determining a passage area. Figure 7 This is a block diagram of a passage area determination device according to an embodiment of this application, such as... Figure 7 As shown, the device 600 for determining the passage area includes: The acquisition module 610 is used to acquire real-time data collected by each of the multiple sensors of the vehicle, wherein the multiple sensors include any combination of image sensors, lidar sensors, and millimeter-wave radar sensors; the identification confidence determination module 620 is used to determine the identification confidence of each of the multiple sensors based on the real-time data; the fusion confidence determination module 630 is used to determine the fusion confidence of the real-time data corresponding to each sensor based on the data type corresponding to the real-time data; and the passable area determination module 640 is used to determine the passable area of the vehicle in the current environment based on the identification confidence of each sensor, the fusion confidence of the real-time data corresponding to each sensor, and the real-time data.
[0096] In some embodiments, the identification confidence determination module 620 includes: a matching degree determination submodule, configured to match the real-time data collected by each sensor with each sample data in the sample database to determine the matching degree corresponding to the real-time data of each sensor; and an identification confidence determination submodule, configured to determine the identification confidence degree of each of the plurality of sensors based on the matching degree.
[0097] In some embodiments, the fusion confidence determination module 630 includes: a data feature determination submodule, configured to determine the data features of real-time data of each sensor corresponding to the data type, wherein the data type includes real-time image data, real-time first point cloud data, and real-time second point cloud data, and the data features include image signal-to-noise ratio, point cloud signal integrity, and spatial coverage; and a fusion confidence determination submodule, configured to determine the fusion confidence corresponding to the real-time data of each sensor based on the data features, wherein the fusion confidence represents the proportion of real-time data indicated by the data features that is fused.
[0098] In some embodiments, the fusion confidence determination submodule includes: a first fusion confidence determination unit, configured to determine a first fusion confidence of the real-time image data based on the image signal-to-noise ratio; a second fusion confidence determination unit, configured to determine a second fusion confidence of the first real-time point cloud data based on the point cloud signal integrity; and a third fusion confidence determination unit, configured to determine a third fusion confidence of the second real-time point cloud data based on the spatial coverage.
[0099] In some embodiments, the sum of the confidence levels of the first fusion confidence level, the second fusion confidence level, and the third fusion confidence level is equal to a confidence threshold.
[0100] In some embodiments, the passable area determination module 640 includes: a reference passable area determination submodule, configured to determine multiple reference passable areas based on real-time data from each sensor; and a passable area determination submodule, configured to fuse the multiple reference passable areas based on the identification confidence and the fusion confidence to obtain the passable area.
[0101] According to one aspect of the embodiments of this application, an electronic device is also provided, such as... Figure 8 As shown, the electronic device 700 also includes a processor 710 and one or more memories 720. The one or more memories 720 are used to store program instructions executed by the processor 710. When the processor 710 executes the program instructions, it implements the above-described method for determining the passable area.
[0102] Furthermore, the processor 710 may include one or more processing cores. The processor 710 runs or executes instructions, programs, code sets, or instruction sets stored in the memory 720, and retrieves data stored in the memory 720. Optionally, the processor 710 may be implemented using at least one hardware form selected from Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The processor 710 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor and may be implemented using a separate communication chip.
[0103] According to one aspect of this application, a computer-readable storage medium is also provided, which may be included in the cloud server described in the above embodiments; or it may exist independently and not assembled into the cloud server. The aforementioned computer-readable storage medium carries computer-readable instructions that, when executed by a processor, implement the methods in any of the above embodiments.
[0104] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. Computer-readable storage media can be, for example, but not limited to: electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such transmitted data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.
[0105] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.
[0106] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.
[0107] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.
Claims
1. A method for determining a passable area, characterized in that, The method includes: The system acquires real-time data collected by each of multiple sensors in the vehicle, wherein the multiple sensors include any combination of image sensors, lidar sensors, and millimeter-wave radar sensors. The identification confidence level of each of the plurality of sensors is determined based on the real-time data; Based on the data type corresponding to the real-time data, determine the fusion confidence level corresponding to the real-time data of each sensor in the real-time data; Based on the identification confidence level of each sensor, the fusion confidence level corresponding to the real-time data of each sensor, and the real-time data, the passable area of the vehicle in the current environment is determined.
2. The method according to claim 1, characterized in that, The step of determining the recognition confidence level of each of the plurality of sensors based on the real-time data includes: The real-time data collected by each sensor is matched with each sample data in the sample database to determine the matching degree corresponding to the real-time data of each sensor. The identification confidence level of each of the plurality of sensors is determined based on the matching degree.
3. The method according to claim 1, characterized in that, The step of determining the fusion confidence level of real-time data for each sensor in the real-time data based on the data type corresponding to the real-time data includes: Determine the data characteristics of real-time data for each sensor corresponding to the data type, wherein the data type includes real-time image data, real-time first point cloud data, and real-time second point cloud data, and the data characteristics include image signal-to-noise ratio, point cloud signal integrity, and spatial coverage; The fusion confidence level corresponding to the real-time data of each sensor is determined based on the data features, wherein the fusion confidence level represents the proportion of real-time data indicated by the data features that are fused.
4. The method according to claim 3, characterized in that, The step of determining the fusion confidence level corresponding to the real-time data of each sensor based on the data features includes: The first fusion confidence level of the real-time image data is determined based on the image signal-to-noise ratio. The second fusion confidence level of the first point cloud real-time data is determined based on the completeness of the point cloud signal. The third fusion confidence level of the second point cloud real-time data is determined based on the spatial coverage.
5. The method according to claim 4, characterized in that, The sum of the confidence levels of the first fusion confidence level, the second fusion confidence level, and the third fusion confidence level is equal to the confidence threshold.
6. The method according to any one of claims 1-5, characterized in that, The step of determining the passable area of the vehicle in the current environment based on the recognition confidence level of each sensor, the fusion confidence level corresponding to the real-time data of each sensor, and the real-time data of each sensor includes: Multiple reference passage areas are determined based on the real-time data from each sensor; The multiple reference passable areas are fused based on the identification confidence and the fusion confidence to obtain the passable area.
7. A device for determining a passable area, characterized in that, The device includes: The acquisition module is used to acquire real-time data collected by each of the multiple sensors of the vehicle, wherein the multiple sensors include any combination of image sensors, lidar sensors and millimeter-wave radar sensors; The identification confidence determination module is used to determine the identification confidence of each of the plurality of sensors based on the real-time data; The fusion confidence determination module is used to determine the fusion confidence of the real-time data of each sensor in the real-time data based on the data type corresponding to the real-time data. The passable area determination module is used to determine the passable area of the vehicle in the current environment based on the identification confidence of each sensor, the fusion confidence of the real-time data of each sensor, and the real-time data.
8. A vehicle, characterized in that, The vehicles include: Multiple sensors, wherein the multiple sensors include any combination of image sensors, lidar sensors and millimeter-wave radar sensors; A processing module is configured to receive data collected by the plurality of sensors and execute the processing actions in the method according to any one of claims 1 to 6.
9. An electronic device, characterized in that, The electronic device includes: processor; A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1 to 6.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor to execute the method as described in any one of claims 1 to 6.