An automatic braking method, apparatus, system, and vehicle

Abstract

The application provides an automatic braking method, device, system and vehicle, wherein the automatic braking method obtains a first image, a second image and radar point cloud data of a target area, the first image is used to represent temperature distribution of the target area, the second image is used to represent two-dimensional visual features of the target area, first data and second data are determined based on the first image, third data is determined based on the second image, and fourth data is determined based on the radar point cloud data, the data categories of the first data and the third data are the same, and the data categories of the second data and the fourth data are the same, target data including the first data, the second data, the third data and the fourth data is used to determine obstacle information of the target area based on the target data, the obstacle information is used to generate an automatic braking instruction for controlling braking of a target vehicle, so that in different scenes, the obstacle information of the target area can be accurately determined, and the robustness of automatic braking is ensured.

Description

Technical Field

[0001] This application relates to the field of vehicle technology, and more specifically, to an automatic braking method, device, system, and vehicle. Background Technology

[0002] Vehicle safety systems can be divided into active safety systems and passive safety systems. Through long-term development in the automotive industry, passive safety technology has become increasingly sophisticated, while active safety technology, centered on preventing dangerous accidents, has become an important direction for the development of modern automotive technology. Automatic Emergency Braking (AEB) is an active safety technology that detects vehicles, pedestrians, or other obstacles that may collide with the vehicle and takes measures such as warning, braking, or evasive maneuvers to avoid a collision, significantly improving driving safety.

[0003] However, existing AEB systems cannot accurately detect obstacles, thus failing to guarantee the robustness of AEB systems. Summary of the Invention

[0004] To address the aforementioned technical problems, this application provides an automatic braking method, device, system, and vehicle to achieve accurate obstacle detection.

[0005] To achieve the above technical objectives, the embodiments of this application provide the following technical solutions:

[0006] Firstly, this specification provides an automatic braking method, comprising:

[0007] Acquire a first image, a second image, and radar point cloud data of the target area; the first image is used to characterize the temperature distribution of the target area, and the second image is used to characterize the two-dimensional visual features of the target area;

[0008] First data and second data are determined based on the first image, third data is determined based on the second image, and fourth data is determined based on the radar point cloud data, wherein the first data and the third data have the same data category, and the second data and the fourth data have the same data category;

[0009] Obstacle information of the target area is determined based on the target data. The obstacle information is used to generate an automatic braking command to control the braking of the target vehicle. The target data includes the first data, the second data, the third data, and the fourth data.

[0010] Optionally, the data categories of the first data and the third data include the semantic features of pixels, and the data categories of the second data and the fourth data include the depth information of pixels;

[0011] The obstacle information for determining the target area based on the target data includes:

[0012] The target semantic features of each pixel in the first image are determined based on the first data and the third data, and the target depth of each pixel in the first image is determined based on the second data and the fourth data.

[0013] Based on the target semantic features of each pixel in the first image, the target depth of each pixel in the first image, and the parameter information of the acquisition device corresponding to the first image, the target point cloud data of the target region is obtained.

[0014] Obstacle information for the target area is determined based on the target point cloud data.

[0015] Optionally, determining the target semantic features of each pixel in the first image based on the first data and the third data includes:

[0016] Based on the correspondence between each pixel in the first image and each pixel in the second image, the semantic features of each pixel in the first data and the semantic features of each pixel in the third data are fused to obtain the target semantic features of each pixel in the first image.

[0017] Optionally, determining the target depth of each pixel in the first image based on the second data and the fourth data includes:

[0018] Based on the depth information of each data point in the fourth data, the depth information of each pixel in the second data is corrected to obtain the target depth of each pixel in the first image.

[0019] Optionally, the data categories of the first data and the third data include obstacle categories, and the data categories of the second data and the fourth data include obstacle distances, wherein the obstacle distances include the distances between the obstacles and the target vehicle;

[0020] The obstacle information for determining the target area based on the target data includes:

[0021] Based on the pixel distance between obstacles and / or the target attribute information of obstacles, a matching process is performed on the first obstacle corresponding to the first image, the second obstacle corresponding to the second image, and the third obstacle corresponding to the radar point cloud data to obtain the target obstacles in the target area; the target attributes include the obstacle category and the obstacle distance;

[0022] Based on the first data and the third data, the target obstacle category of the target obstacle is determined, and based on the second data and the fourth data, the target obstacle distance of the target obstacle is determined.

[0023] Optionally, determining the target obstacle category based on the first data and the third data includes:

[0024] Based on the obstacle category of the target obstacle in the first data, the obstacle category of the target obstacle in the third data, the priority of the first data, and the priority of the third data, the target obstacle category of the target obstacle is determined;

[0025] And / or, determining the target obstacle distance based on the second data and the fourth data includes:

[0026] The target obstacle distance of the target obstacle is determined based on the obstacle distance of the target obstacle in the second data, the obstacle distance of the target obstacle in the fourth data, the priority of the second data, and the priority of the fourth data.

[0027] Optionally, after determining the obstacle information of the target area based on the target data, the method further includes:

[0028] The obstacle information and the target vehicle's driving information are input into a pre-trained decision model, and the decision model outputs an automatic braking command for the target vehicle.

[0029] The decision model is trained based on a preset loss function, which includes the loss due to the false trigger rate of automatic braking.

[0030] Secondly, embodiments of this specification provide an automatic braking device, including:

[0031] The first processing module is used to acquire a first image, a second image, and radar point cloud data of the target area; the first image is used to characterize the temperature distribution of the target area, and the second image is used to characterize the two-dimensional visual features of the target area.

[0032] The second processing module is used to determine first data and second data based on the first image, determine third data based on the second image, and determine fourth data based on the radar point cloud data, wherein the first data and the third data have the same data category, and the second data and the fourth data have the same data category.

[0033] The third processing module is used to determine obstacle information of the target area based on the target data. The obstacle information is used to generate an automatic braking command to control the braking of the target vehicle. The target data includes the first data, the second data, the third data, and the fourth data.

[0034] Thirdly, this specification provides an automatic braking system, including: a data acquisition unit and a control unit;

[0035] The data acquisition unit is used to acquire a first image, a second image, and radar point cloud data of the target area; the first image is used to characterize the temperature distribution of the target area, and the second image is used to characterize the two-dimensional visual features of the target area.

[0036] The control unit is connected to the data acquisition unit and is used to determine first data and second data based on the first image, determine third data based on the second image, and determine fourth data based on the radar point cloud data. The first data and the third data have the same data category, and the second data and the fourth data have the same data category. The control unit also determines obstacle information of the target area based on the target data. The obstacle information is used to generate an automatic braking command to control the braking of the target vehicle. The target data includes the first data, the second data, the third data, and the fourth data.

[0037] Fourthly, embodiments of this specification provide a vehicle that includes the automatic braking system described above.

[0038] Fifthly, embodiments of this specification provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the automatic braking method as described in any of the preceding claims.

[0039] Sixthly, embodiments of this specification provide a computer program product or computer program, the computer program product including a computer program stored in a computer-readable storage medium; a processor of the computer device reads the computer program from the computer-readable storage medium, and when the processor executes the computer program, it implements the automatic braking method as described in any of the preceding claims.

[0040] As can be seen from the above technical solutions, the embodiments of this application provide an automatic braking method, device, system, and vehicle. The automatic braking method acquires a first image, a second image, and radar point cloud data of a target area. The first image characterizes the temperature distribution of the target area, and the second image characterizes the two-dimensional visual features of the target area. Based on the first image, first data and second data are determined; based on the second image, third data is determined; and based on the radar point cloud data, fourth data is determined. The first and third data have the same data category, and the second and fourth data have the same data category, meaning the first and third data are redundant, and the second and fourth data are also redundant. This allows for the determination of obstacle information in the target area based on the target data. The obstacle information is used to generate an automatic braking command to control the braking of the target vehicle. The target data includes the first, second, third, and fourth data. Therefore, in different scenarios, when some data in the first image, second image, and radar point cloud data is missing or has low accuracy, the redundancy design ensures the comprehensiveness and accuracy of the target data. This allows for the accurate determination of obstacle information in the target area based on the target data, ensuring the robustness of the automatic braking. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0042] Figure 1 A flowchart illustrating an automatic braking method provided for one embodiment of this specification;

[0043] Figure 2 A flowchart illustrating a method for determining obstacle information provided in one embodiment of this specification;

[0044] Figure 3 A flowchart illustrating another method for determining obstacle information provided in one embodiment of this specification;

[0045] Figure 4 A schematic diagram of an automatic braking device provided for one embodiment of this specification;

[0046] Figure 5 This is a schematic diagram of an automatic braking system provided for one embodiment of this specification. Detailed Implementation

[0047] Unless otherwise defined, the technical or scientific terms used in the embodiments of this specification shall have the ordinary meaning understood by one of ordinary skill in the art to which this specification pertains. The terms "first," "second," and similar terms used in the embodiments of this specification do not indicate any order, quantity, or importance, but are merely used to avoid confusion of constituent elements.

[0048] Unless the context otherwise requires, throughout this specification, "a plurality of" means "at least two," and "including" is interpreted as open-ended or encompassing, that is, "including, but not limited to." In the description of this specification, terms such as "one embodiment," "some embodiments," "exemplary embodiment," "example," "specific example," or "some examples" are intended to indicate that a particular feature, structure, material, or characteristic associated with that embodiment or example is included in at least one embodiment or example of this specification. The illustrative representations of the above terms do not necessarily refer to the same embodiment or example.

[0049] The technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.

[0050] Overview

[0051] Vehicle safety systems can be divided into active safety systems and passive safety systems. Through long-term development in the automotive industry, passive safety technology has become increasingly sophisticated, while active safety technology, centered on preventing dangerous accidents, has become an important direction for the development of modern automotive technology. Automatic Emergency Braking (AEB) is an active safety technology that detects vehicles, pedestrians, or other obstacles that may collide with the vehicle and takes measures such as warning, braking, or evasive maneuvers to avoid a collision, significantly improving driving safety.

[0052] The AEB (Autonomous Emergency Braking) system mainly consists of three parts: a driving environment information acquisition unit (sensors), an electronic control unit (central processing unit), and an execution unit (actuator). Its primary task is to accurately detect the driving environment in real time through the driving environment information acquisition unit to obtain relevant driving information. Currently, the mainstream detection methods are camera and radar detection: cameras identify the type of obstacle, and radar senses the distance, speed, and azimuth of the obstacle. However, this approach has certain limitations: poor nighttime lighting and adverse weather conditions (rain, snow, fog, dust, etc.) can significantly degrade system performance, and the recognition rate for small targets such as pedestrians and cyclists is not high. Furthermore, there are issues such as false recognition, which compromises the robustness of the AEB system.

[0053] To address the problem of traditional methods failing to accurately detect obstacles during automatic braking, this application employs a method that acquires a first image, a second image, and radar point cloud data of the target area. The first image characterizes the temperature distribution of the target area, and the second image characterizes its two-dimensional visual features. Based on the first image, first and second data are determined; based on the second image, third data is determined; and based on the radar point cloud data, fourth data is determined. The first and third data belong to the same data category, and the second and fourth data belong to the same data category, meaning they are redundant. This allows for the determination of obstacle information in the target area based on the target data. The obstacle information is used to generate automatic braking commands to control the braking of the target vehicle. The target data includes the first, second, third, and fourth data. Thus, even in different scenarios where some data in the first image, second image, and radar point cloud data is missing or has low accuracy, the redundancy design ensures the comprehensiveness and accuracy of the target data. Consequently, the obstacle information in the target area can be accurately determined based on the target data, guaranteeing the robustness of automatic braking.

[0054] Furthermore, the automatic braking method provided in this specification also specifies the method for determining obstacle information in the target area, so as to quickly and accurately detect obstacles in the target area.

[0055] Based on the above inventive concept, the automatic braking method provided in the embodiments of this specification will be described exemplarily below.

[0056] Exemplary methods

[0057] This specification provides an automatic braking method, such as... Figure 1 As shown, it includes:

[0058] S101. Acquire a first image, a second image, and radar point cloud data of the target area; the first image is used to characterize the temperature distribution of the target area, and the second image is used to characterize the two-dimensional visual features of the target area.

[0059] Specifically, the target area is the area where the target vehicle is traveling, and the target vehicle can be a vehicle that needs to be automatically braked.

[0060] The first image of the target area is used to characterize the temperature distribution of the target area. For example, the first image can be a thermal infrared image. Thus, in scenes with poor visual conditions, and for smaller obstacles such as pedestrians and cyclists, the temperature distribution of the target area can be used to accurately detect obstacles. The first image can be acquired by an infrared thermal imager mounted on the target vehicle.

[0061] The second image of the target area is used to characterize the two-dimensional visual features of the target area. These features can include color, texture, shape, and other characteristics, enabling accurate obstacle detection in scenarios with good visual conditions. For example, the second image can be an RGB (Red, Green, Blue) image. This second image can be acquired using a camera mounted on the target vehicle.

[0062] Radar point cloud data of the target area can be collected by radar installed on the target vehicle. For example, millimeter-wave radar can be used. Millimeter-wave radar has a longer detection range, so the radar point cloud data collected by millimeter-wave radar can identify obstacles at a far distance in advance, thus allowing sufficient time to make automatic braking decisions and effectively improving the performance of automatic braking.

[0063] S102. Determine first data and second data based on the first image, determine third data based on the second image, and determine fourth data based on the radar point cloud data, wherein the first data and the third data have the same data category, and the second data and the fourth data have the same data category.

[0064] Specifically, the first and third data have the same data category, and the second and fourth data have the same data category. That is, the first and third data can be redundant data, and the second and fourth data can be redundant data.

[0065] For example, the data categories for the first and third data can both be semantic features of pixels, while the data categories for the second and fourth data can both be depth information of pixels. Additionally, the data categories for the first and third data can both be obstacle categories, and the data categories for the second and fourth data can both be obstacle distances; these can be set according to actual needs.

[0066] In practice, after acquiring the first image, the second image, and radar point cloud data of the target area, data processing can be performed on the first image, the second image, and the radar point cloud data, and the first data, the second data, the third data, and the fourth data can be determined based on the processed first image, the second image, and the radar point cloud data.

[0067] For example, distortion correction can be performed on the first image based on the intrinsic parameters of the acquisition device corresponding to the first image, and the distortion correction can be performed on the second image based on the intrinsic parameters of the acquisition device corresponding to the second image. Simultaneously, time synchronization processing can be performed on the first image, the second image, and the radar point cloud data. Furthermore, the first image, the second image, and the radar point cloud data can be projected onto the same coordinate system based on pre-calibrated extrinsic parameters to perform pixel alignment. It is understood that the pre-calibrated extrinsic parameters can be used to characterize the relative positions and attitudes between the acquisition devices corresponding to the first image, the second image, and the radar point cloud data.

[0068] S103. Determine obstacle information of the target area based on the target data. The obstacle information is used to generate an automatic braking command to control the braking of the target vehicle. The target data includes the first data, the second data, the third data, and the fourth data.

[0069] Specifically, the target data may include first data, second data, third data, and fourth data. In addition, the target data may also include data other than the fourth data detected by radar point cloud data. For example, the target data may also include information such as the velocity and azimuth of each data point in the radar point cloud data, or the target data may also include the obstacle velocity and obstacle azimuth of the third obstacle corresponding to the radar point cloud data.

[0070] In the process of determining obstacle information of the target area based on target data, the first sub-target data of the corresponding data category can be determined from the first data and the third data based on the priority of the first data and the priority of the third data. In addition, feature fusion can be performed on the first data and the third data to obtain the first sub-target data of the corresponding data category.

[0071] At the same time, based on the priority of the second data and the priority of the fourth data, the second sub-target data of the corresponding data category can be determined from the second data and the fourth data. In addition, feature fusion can be performed on the second data and the fourth data to obtain the second sub-target data of the corresponding data category.

[0072] After determining the first and second sub-target data, obstacle information in the target area can be determined based on these data. Therefore, in different scenarios, even when some data in the first image, second image, and radar point cloud data is missing or has low accuracy, redundancy design can still ensure the comprehensiveness and accuracy of the target data. This allows for accurate determination of obstacle information in the target area based on the target data, thus ensuring the robustness of automatic braking.

[0073] In practice, automatic braking commands can be generated based on obstacle information in the target area to control the target vehicle to brake. It is understood that automatic braking commands can also be generated based on both obstacle information and the target vehicle's driving information simultaneously.

[0074] To further improve the comprehensiveness and accuracy of the obstacle information determination results in the target area, in one embodiment of this specification, the data categories of the first data and the third data include the semantic features of pixels, and the data categories of the second data and the fourth data include the depth information of pixels;

[0075] like Figure 2 As shown, determining the obstacle information of the target area based on the target data includes:

[0076] S201. Determine the target semantic features of each pixel in the first image based on the first data and the third data, and determine the target depth of each pixel in the first image based on the second data and the fourth data.

[0077] S202. Based on the target semantic features of each pixel in the first image, the target depth of each pixel in the first image, and the parameter information of the acquisition device corresponding to the first image, obtain the target point cloud data of the target region.

[0078] S203. Determine the obstacle information of the target area based on the target point cloud data.

[0079] Specifically, the data categories of the first and third data can include semantic features of pixels, while the data categories of the second and fourth data can include depth information of pixels. Semantic features can be used to characterize the attribute information of the corresponding pixels; for example, attribute information can include color features, texture features, shape features, etc. Depth information can be used to characterize the distance or depth value of the corresponding pixel relative to the target vehicle.

[0080] In implementation, semantic feature extraction can be performed on the first image to obtain first data, which includes the semantic features of each pixel in the first image. Simultaneously, depth estimation can be performed on each pixel in the first image using a depth estimation network to obtain second data, which includes the depth information of each pixel in the first image. The depth estimation network can be a convolutional neural network, a residual network, an attention mechanism network, etc. Furthermore, semantic feature extraction can be performed on the second image to obtain third data, which includes the semantic features of each pixel in the second image. Additionally, depth information of each data point in the radar point cloud data can be obtained based on the attribute information of each data point, resulting in fourth data. It is understandable that information such as velocity and azimuth of each data point in the radar point cloud data can also be obtained based on the attribute information of each data point.

[0081] In the process of determining the target semantic features of each pixel in the first image based on the first data and the third data, for any pixel in the first image, the semantic features of the pixel in the data with higher priority in the first data and the third data, or the semantic features of the pixel corresponding to the pixel, can be used as the target semantic features of the pixel based on the priority of the first data and the priority of the third data. Alternatively, the semantic features of the pixel in the first data and the semantic features of the pixel corresponding to the pixel in the third data can be fused to obtain the target semantic features of the pixel.

[0082] Furthermore, in determining the target depth of each pixel in the first image based on the second and fourth data, for any pixel in the first image, the target depth can be determined based on the depth information of that pixel in the higher-priority data of the second and fourth data, or the depth information of the data point corresponding to the region where that pixel is located. It is understood that the number of pixels in the first image is greater than the number of data points in the radar point cloud data, and one data point in the radar point cloud data can correspond to one region in the first image.

[0083] After determining the target semantic features and target depth of each pixel in the first image, point cloud data can be obtained by fusing features from the first image, the second image, and radar point cloud data, based on the target semantic features and target depth of each pixel in the first image. This fused point cloud data serves as the fused point cloud data, meaning that each data point in the fused point cloud data corresponds one-to-one with each pixel in the first image. It is understandable that information such as velocity and azimuth angle of each data point in the radar point cloud data can also be added to the corresponding pixels in the first image to ensure the comprehensiveness of the features in the fused point cloud data.

[0084] The parameter information of the acquisition device corresponding to the first image can include the correspondence between the camera coordinate system and the world coordinate system of the acquisition device corresponding to the first image. In practice, the coordinate system can be transformed on the fused point cloud data based on the correspondence between the camera coordinate system and the world coordinate system of the acquisition device corresponding to the first image to obtain the target point cloud data of the target area.

[0085] After obtaining the target point cloud data of the target area, obstacle information can be determined based on this data. Obstacle information can include obstacles and their attributes, such as obstacle category, distance, speed, and azimuth. For example, a 3D point cloud detection network can be used to detect obstacles and their attributes within the target point cloud data. Because each data point in the target point cloud data corresponds one-to-one with a pixel in the first image, the target point cloud data is denser than radar point cloud data. This allows for a more complete and accurate representation of obstacles, thus improving the accuracy of obstacle detection. Furthermore, the attributes of each data point in the target point cloud data integrate features from the first image, the second image, and the radar point cloud data. Through the complementary and optimized combination of features from these three images, a consistent interpretation of obstacle information can be generated, further improving the accuracy of obstacle detection.

[0086] In one feasible implementation, determining the target semantic features of each pixel in the first image based on the first data and the third data includes:

[0087] Based on the correspondence between each pixel in the first image and each pixel in the second image, the semantic features of each pixel in the first data and the semantic features of each pixel in the third data are fused to obtain the target semantic features of each pixel in the first image.

[0088] Specifically, in the process of determining the target semantic features of each pixel in the first image based on the first data and the third data, the semantic features of each pixel in the first data and the semantic features of each pixel in the third data can be fused based on the correspondence between each pixel in the first image and each pixel in the second image to obtain the target semantic features of each pixel in the first image.

[0089] For example, for any pixel in the first image, the semantic features of that pixel in the first data and the semantic features of the corresponding pixel in the third data can be fused to obtain the target semantic features of that pixel. Feature fusion can be feature concatenation, that is, the target semantic features of that pixel simultaneously include the semantic features of that pixel in the first data and the semantic features of the corresponding pixel in the third data, thereby effectively ensuring the comprehensiveness of the target semantic features of each pixel in the first image.

[0090] One approach is to introduce an attention mechanism into the 3D point cloud detection network. This mechanism automatically adjusts the weights of semantic features of pixels in the first and third datasets of the target point cloud data, effectively improving the accuracy of obstacle detection. Furthermore, the automatic weight adjustment via the 3D point cloud detection network eliminates the need for detection and verification of results from different data sources, significantly increasing the efficiency of obstacle detection.

[0091] In one feasible implementation, determining the target depth of each pixel in the first image based on the second data and the fourth data includes:

[0092] Based on the depth information of each data point in the fourth data, the depth information of each pixel in the second data is corrected to obtain the target depth of each pixel in the first image.

[0093] Specifically, the accuracy of depth information of data points in radar point cloud data is usually higher than that of depth information of pixels in thermal infrared images. Therefore, the depth information of each pixel in the second data can be corrected based on the depth information of each pixel in the fourth data to obtain the target depth of each pixel in the first image.

[0094] In implementation, for any data point in the radar point cloud data, the depth information of pixels in the target region of the first image can be corrected based on the depth information of that data point. For example, when the depth information of the data point differs from the depth information of the first pixel in the target region, the depth information of the first pixel can be adjusted to match the depth information of the data point. Simultaneously, based on the difference between the depth information of the first pixel and the data point, the depth information of the second pixel in the target region is corrected. That is, the corrected depth information of the second pixel is the sum of the depth information of the second pixel in the second data and the difference. Here, the target region can be the region in the first image corresponding to the data point, the first pixel can be the pixel in the target region with the smallest pixel distance to the data point, and the second pixel can be any pixel in the target region other than the first pixel.

[0095] Therefore, by correcting the depth information of each pixel in the first image using the depth information of each data point in the fourth data, the accuracy of the determination result of the target depth of each pixel in the first image can be effectively guaranteed.

[0096] It is understandable that if the radar point cloud data acquisition device malfunctions, the target depth of each pixel in the first image can be directly adopted from the depth information of each pixel in the second data, thereby ensuring fault tolerance during the automatic braking process.

[0097] To further improve the comprehensiveness and accuracy of the determination results of obstacle information in the target area, in one embodiment of this specification, the data categories of the first data and the third data include obstacle categories, and the data categories of the second data and the fourth data include obstacle distances, wherein the obstacle distances include the distances between the obstacles and the target vehicle;

[0098] like Figure 3 As shown, determining the obstacle information of the target area based on the target data includes:

[0099] S301. Based on the pixel distance between obstacles and / or the target attribute information of obstacles, perform matching processing on the first obstacle corresponding to the first image, the second obstacle corresponding to the second image, and the third obstacle corresponding to the radar point cloud data to obtain the target obstacles in the target area; the target attributes include the obstacle category and the obstacle distance.

[0100] S302. Based on the first data and the third data, determine the target obstacle category of the target obstacle, and based on the second data and the fourth data, determine the target obstacle distance of the target obstacle.

[0101] Specifically, the data categories for the first and third data can include obstacle categories, while the data categories for the second and fourth data can include obstacle distances. Obstacle categories can include vehicles, pedestrians, traffic signs, and buildings, etc. Obstacle distances can include the distance between the obstacle and the target vehicle.

[0102] In implementation, obstacle detection can be performed on the first image using a thermal infrared image sensing method to obtain the first obstacles corresponding to the first image and the obstacle categories of each first obstacle. That is, the first data includes the obstacle categories of each first obstacle corresponding to the first image. The thermal infrared image sensing method can employ a machine learning model to detect obstacles in the first image; the machine learning model can include support vector machine models, convolutional neural network models, etc. Simultaneously, the obstacle distances of each first obstacle corresponding to the first image can be detected using a thermal infrared image ranging method. That is, the second data includes the obstacle distances of each first obstacle corresponding to the first image.

[0103] Additionally, obstacle detection can be performed on the second image using an RGB image perception method to obtain the second obstacles corresponding to the second image and the obstacle category of each second obstacle. That is, the third data includes the obstacle category of each second obstacle corresponding to the second image. The RGB image perception method can employ a machine learning model to detect obstacles in the second image; this machine learning model can include support vector machine models, convolutional neural network models, etc.

[0104] Meanwhile, obstacle detection can be performed on radar point cloud data using point cloud perception algorithms to obtain the third obstacle corresponding to the radar point cloud data, as well as the obstacle distance, obstacle speed, obstacle azimuth angle, etc. of each third obstacle. That is, the fourth data includes the obstacle distance of each third obstacle corresponding to the radar point cloud data.

[0105] In the process of determining obstacle information of the target area based on target data, the first obstacle, the second obstacle, and the third obstacle can be matched based on the pixel distance between obstacles and / or the target attribute information of obstacles. The pixel distance is the distance in pixels, and the target attribute information of obstacles can include obstacle category and obstacle distance.

[0106] Specifically, by performing pixel alignment on the first image, the second image, and the radar point cloud data, the pixel distance between the first obstacle, the second obstacle, and the third obstacle can be determined based on any one of the data from the first image, the second image, and the radar point cloud data.

[0107] Taking the first image as an example, the pixel distance between the second obstacle and the first obstacle can be the pixel distance between the target point of the first obstacle and the target point of the second obstacle in the first image. The target point of the second obstacle can be the pixel in the first image that corresponds to the target pixel. The target pixel is the pixel in the second image that corresponds to the target point of the second obstacle. The target point can be the center point of the obstacle. Similarly, the pixel distance between the third obstacle and the first obstacle can be the pixel distance between the target point of the first obstacle and the target point of the third obstacle in the first image. The target point of the third obstacle can be the pixel in the first image that corresponds to the target data point. The target data point is the data point in the radar point cloud data that corresponds to the target point of the third obstacle.

[0108] In practice, the first obstacle, the second obstacle, and the third obstacle can be matched using the Hungarian matching method based on the pixel distance between the obstacles to obtain the matching results of the first obstacle, the second obstacle, and the third obstacle.

[0109] In addition, the first obstacle, the second obstacle, and the third obstacle can be matched based on the target attribute information of the obstacles. For example, the second obstacle can be matched with the first obstacle based on the obstacle category, and the third obstacle can be matched with the first obstacle based on the obstacle distance, so as to obtain the matching results of the first obstacle, the second obstacle, and the third obstacle.

[0110] Understandably, the first, second, and third obstacles can also be matched simultaneously based on the pixel distance between obstacles and the target attribute information of the obstacles. For example, the first, second, and third obstacles can be matched based on the pixel distance between obstacles to obtain an initial matching result, and the initial matching result can be corrected based on the target attribute information of the obstacles to obtain the matching processing result of the first, second, and third obstacles. Thus, when the pixel distance between obstacles is similar, the matching processing result of the first, second, and third obstacles can be obtained quickly and accurately based on the correspondence between obstacle category and obstacle distance in the first image.

[0111] After obtaining the matching results of the first obstacle, the second obstacle, and the third obstacle, the target obstacle in the target area can be determined based on the matching results. For example, the intersection of the first obstacle, the second obstacle, and the third obstacle can be used as the target obstacle. That is, the target obstacle is the obstacle detected in the first image, the second image, and the radar point cloud data. Alternatively, the intersection of the first obstacle and the third obstacle, as well as the intersection of the second obstacle and the third obstacle, can be used as the target obstacle to ensure the integrity of the attribute information of the target obstacle.

[0112] After identifying target obstacles within the target area, the obstacle category can be determined based on the first and third data. Simultaneously, the obstacle distance can be determined based on the second and fourth data. For example, if the obstacle category exists in both the first and third data for any given obstacle, the obstacle category with the higher priority among the first and third data can be used as the target obstacle category. Similarly, if the obstacle distance exists in both the second and fourth data, the obstacle distance with the higher priority among the second and fourth data can be used as the target obstacle distance. This effectively ensures the accuracy of the obstacle attribute information determination results.

[0113] It is understandable that obstacle information in the target area can include the target obstacle and its attribute information. The attribute information of the target obstacle can include the target obstacle category and the distance to the target obstacle. In addition, the attribute information of the target obstacle can also include obstacle speed, obstacle azimuth angle, etc., which are detected by radar point cloud data.

[0114] In one feasible implementation, determining the target obstacle category based on the first data and the third data includes:

[0115] Based on the obstacle category of the target obstacle in the first data, the obstacle category of the target obstacle in the third data, the priority of the first data, and the priority of the third data, the target obstacle category of the target obstacle is determined;

[0116] And / or, determining the target obstacle distance based on the second data and the fourth data includes:

[0117] The target obstacle distance of the target obstacle is determined based on the obstacle distance of the target obstacle in the second data, the obstacle distance of the target obstacle in the fourth data, the priority of the second data, and the priority of the fourth data.

[0118] Specifically, the priorities of the first, second, third, and fourth data can be adjusted in real time based on environmental information and the operating status of the acquisition devices for the first image, second image, and radar point cloud data. When an acquisition device is malfunctioning, the priority of the data corresponding to that malfunctioning acquisition device is 0. When the acquisition devices for the first image, second image, and radar point cloud data are all operating normally, the priority of the fourth data is higher than that of the second data. Furthermore, in environments with poor lighting at night or severe weather conditions (rain, snow, fog, dust, etc.), the priority of the first data is higher than that of the third data; otherwise, the priority of the third data is higher than that of the first data.

[0119] Furthermore, in the first data set, the priority of the obstacle category for each first obstacle can be different. Similarly, in the third data set, the priority of the obstacle category for each second obstacle can also be different. The priority of the corresponding obstacle category in the first data set and the corresponding obstacle category in the third data set can be determined based on the size of the target obstacle. For example, for smaller obstacles such as pedestrians and cyclists, the priority of the corresponding obstacle category in the first data set is higher than the priority of the corresponding obstacle category in the third data set; for larger obstacles, the priority of the corresponding obstacle category in the third data set is higher than the priority of the corresponding obstacle category in the first data set.

[0120] In implementation, different weights can be set for different priority influencing factors. Based on the weights and the priorities of each priority influencing factor, the priorities of the first, second, third, and fourth data are determined by weighted summation. The obstacle category of the target obstacle in the first and third data with higher priority is taken as the target obstacle category of the target obstacle, and the obstacle distance of the target obstacle in the second and fourth data with higher priority is taken as the target obstacle distance of the target obstacle. This further improves the accuracy of the obstacle information determination results in the target area. At the same time, there is no need to detect and verify the detection results from different data sources, which can greatly improve the detection efficiency of obstacle information.

[0121] To ensure a balance between obstacle detection accuracy and automatic braking false trigger rate during automatic braking, in one embodiment of this specification, after determining the obstacle information of the target area based on target data, the method further includes:

[0122] The obstacle information and the target vehicle's driving information are input into a pre-trained decision model, and the decision model outputs an automatic braking command for the target vehicle.

[0123] The decision model is trained based on a preset loss function, which includes the loss due to the false trigger rate of automatic braking.

[0124] Specifically, the target vehicle's driving information can include its speed, steering angle, and road conditions. During implementation, the target vehicle's driving information can be acquired in real time, and obstacle information in the target area, along with the target vehicle's driving information, can be input into a pre-trained decision model. This decision model can be a convolutional neural network model, which automatically outputs automatic braking commands for the target vehicle. These commands can include whether to activate the AEB system, the degree of intervention by the AEB system, and how to adjust the target vehicle's driving state, thereby enabling rapid and accurate automatic braking decisions.

[0125] The decision model can be trained based on a preset loss function, which may include the false trigger rate of automatic braking. It is understood that the higher the detection rate of obstacle information in the target area, the higher the sensitivity of triggering the AEB system based on the decision model's decision results, thus leading to an increased false trigger rate of the AEB system. Therefore, training the decision model based on the false trigger rate loss can effectively ensure a balance between obstacle detection accuracy and the false trigger rate of automatic braking.

[0126] Exemplary devices, systems, and vehicles

[0127] refer to Figure 4 This specification also provides an automatic braking device, comprising:

[0128] The first processing module 401 is used to acquire a first image, a second image, and radar point cloud data of the target area; the first image is used to characterize the temperature distribution of the target area, and the second image is used to characterize the two-dimensional visual features of the target area.

[0129] The second processing module 402 is used to determine first data and second data based on the first image, determine third data based on the second image, and determine fourth data based on the radar point cloud data, wherein the first data and the third data have the same data category, and the second data and the fourth data have the same data category.

[0130] The third processing module 403 is used to determine obstacle information of the target area based on the target data. The obstacle information is used to generate an automatic braking command to control the braking of the target vehicle. The target data includes the first data, the second data, the third data, and the fourth data.

[0131] In one feasible implementation, the data categories of the first data and the third data include semantic features of pixels, and the data categories of the second data and the fourth data include depth information of pixels.

[0132] The third processing module 403 is specifically used for:

[0133] The target semantic features of each pixel in the first image are determined based on the first data and the third data, and the target depth of each pixel in the first image is determined based on the second data and the fourth data.

[0134] Based on the target semantic features of each pixel in the first image, the target depth of each pixel in the first image, and the parameter information of the acquisition device corresponding to the first image, the target point cloud data of the target region is obtained.

[0135] Obstacle information for the target area is determined based on the target point cloud data.

[0136] In one feasible implementation, the third processing module 403 is specifically used for:

[0137] Based on the correspondence between each pixel in the first image and each pixel in the second image, the semantic features of each pixel in the first data and the semantic features of each pixel in the third data are fused to obtain the target semantic features of each pixel in the first image.

[0138] In one feasible implementation, the third processing module 403 is specifically used for:

[0139] Based on the depth information of each data point in the fourth data, the depth information of each pixel in the second data is corrected to obtain the target depth of each pixel in the first image.

[0140] In one feasible implementation, the data categories of the first data and the third data include obstacle categories, and the data categories of the second data and the fourth data include obstacle distances, wherein the obstacle distances include the distances between the obstacles and the target vehicle;

[0141] The third processing module 403 is specifically used for:

[0142] Based on the pixel distance between obstacles and / or the target attribute information of obstacles, a matching process is performed on the first obstacle corresponding to the first image, the second obstacle corresponding to the second image, and the third obstacle corresponding to the radar point cloud data to obtain the target obstacles in the target area; the target attributes include the obstacle category and the obstacle distance;

[0143] Based on the first data and the third data, the target obstacle category of the target obstacle is determined, and based on the second data and the fourth data, the target obstacle distance of the target obstacle is determined.

[0144] In one feasible implementation, the third processing module 403 is specifically used for:

[0145] Based on the obstacle category of the target obstacle in the first data, the obstacle category of the target obstacle in the third data, the priority of the first data, and the priority of the third data, the target obstacle category of the target obstacle is determined;

[0146] And / or,

[0147] The target obstacle distance of the target obstacle is determined based on the obstacle distance of the target obstacle in the second data, the obstacle distance of the target obstacle in the fourth data, the priority of the second data, and the priority of the fourth data.

[0148] In one feasible implementation, a fourth processing module is further included, the fourth processing module being used for:

[0149] The obstacle information and the target vehicle's driving information are input into a pre-trained decision model, and the decision model outputs an automatic braking command for the target vehicle.

[0150] The decision model is trained based on a preset loss function, which includes the loss due to the false trigger rate of automatic braking.

[0151] The automatic braking device provided in this embodiment belongs to the same concept as the automatic braking method provided in the above embodiments of this application. It can execute the automatic braking method provided in any of the above embodiments of this application and has the corresponding functional modules and beneficial effects for executing the automatic braking method. Technical details not described in detail in this embodiment can be found in the specific processing content of the automatic braking method provided in the above embodiments of this application, and will not be repeated here.

[0152] refer to Figure 5 The embodiments of this specification also provide an automatic braking system, including: a data acquisition unit 501 and a control unit 502;

[0153] The data acquisition unit 501 is used to acquire a first image, a second image, and radar point cloud data of the target area; the first image is used to characterize the temperature distribution of the target area, and the second image is used to characterize the two-dimensional visual features of the target area.

[0154] The control unit 502 is connected to the data acquisition unit 501 and is used to determine first data and second data based on the first image, determine third data based on the second image, and determine fourth data based on the radar point cloud data. The first data and the third data have the same data category, and the second data and the fourth data have the same data category. The control unit 502 is also used to determine obstacle information of the target area based on the target data. The obstacle information is used to generate an automatic braking command to control the braking of the target vehicle. The target data includes the first data, the second data, the third data, and the fourth data.

[0155] This specification also provides a vehicle that includes the automatic braking system described in the above embodiments.

[0156] Exemplary computer program products and storage media

[0157] In addition to the methods and devices described above, the automatic braking method provided in the embodiments of this specification can also be a computer program product, which includes computer program instructions that, when executed by a processor, cause the processor to perform the steps in the automatic braking method according to various embodiments of this specification as described in the "Exemplary Methods" section above.

[0158] The computer program product described herein can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments described herein. These programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0159] Furthermore, embodiments of this specification also provide a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor of the steps in the automatic braking method according to various embodiments of this specification as described in the "Exemplary Methods" section above.

[0160] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this specification can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0161] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0162] The embodiments described above are merely illustrative of several implementation methods outlined in this specification. While the descriptions are specific and detailed, they should not be construed as limiting the scope of the solutions provided in this specification. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this specification, and these all fall within the scope of protection of this specification. Therefore, the scope of protection for this patent should be determined by the appended claims.

Claims

1. An automatic braking method, characterized in that, include: Acquire the first image, the second image, and radar point cloud data of the target area; The first image is used to characterize the temperature distribution of the target region, and the second image is used to characterize the two-dimensional visual features of the target region; First data and second data are determined based on the first image, third data is determined based on the second image, and fourth data is determined based on the radar point cloud data. The first data and the third data have the same data category, and the second data and the fourth data have the same data category. The data categories of the first data and the third data include obstacle categories, and the data categories of the second data and the fourth data include obstacle distances, where the obstacle distances include the distance between the obstacle and the target vehicle. Obstacle information of the target area is determined based on the target data. The obstacle information is used to generate an automatic braking command to control the braking of the target vehicle. The target data includes the first data, the second data, the third data, and the fourth data. The step of determining the obstacle information of the target area based on the target data includes: determining the target obstacle category based on the obstacle category of the target obstacle in the first data, the obstacle category of the target obstacle in the third data, the priority of the first data, and the priority of the third data; Furthermore, the target obstacle distance of the target obstacle is determined based on the obstacle distance of the target obstacle in the second data, the obstacle distance of the target obstacle in the fourth data, the priority of the second data, and the priority of the fourth data.

2. The method according to claim 1, characterized in that, The data categories of the first data and the third data include the semantic features of pixels, while the data categories of the second data and the fourth data include the depth information of pixels. The obstacle information for determining the target area based on the target data includes: The target semantic features of each pixel in the first image are determined based on the first data and the third data, and the target depth of each pixel in the first image is determined based on the second data and the fourth data. Based on the target semantic features of each pixel in the first image, the target depth of each pixel in the first image, and the parameter information of the acquisition device corresponding to the first image, the target point cloud data of the target region is obtained. Obstacle information for the target area is determined based on the target point cloud data.

3. The method according to claim 2, characterized in that, Determining the target semantic features of each pixel in the first image based on the first data and the third data includes: Based on the correspondence between each pixel in the first image and each pixel in the second image, the semantic features of each pixel in the first data and the semantic features of each pixel in the third data are fused to obtain the target semantic features of each pixel in the first image.

4. The method according to claim 2, characterized in that, Determining the target depth of each pixel in the first image based on the second data and the fourth data includes: Based on the depth information of each data point in the fourth data, the depth information of each pixel in the second data is corrected to obtain the target depth of each pixel in the first image.

5. The method according to claim 1, characterized in that, The method for determining the target obstacle includes: Based on the pixel distance between obstacles and / or the target attribute information of obstacles, a matching process is performed on the first obstacle corresponding to the first image, the second obstacle corresponding to the second image, and the third obstacle corresponding to the radar point cloud data to obtain the target obstacles in the target area; the target attributes include the obstacle category and the obstacle distance.

6. The method according to any one of claims 1 to 5, characterized in that, After determining the obstacle information of the target area based on the target data, the method further includes: The obstacle information and the target vehicle's driving information are input into a pre-trained decision model, and the decision model outputs an automatic braking command for the target vehicle. The decision model is trained based on a preset loss function, which includes the loss due to the false trigger rate of automatic braking.

7. An automatic braking device, characterized in that, include: The first processing module is used to acquire the first image, the second image, and radar point cloud data of the target area; The first image is used to characterize the temperature distribution of the target region, and the second image is used to characterize the two-dimensional visual features of the target region; The second processing module is configured to determine first data and second data based on the first image, determine third data based on the second image, and determine fourth data based on the radar point cloud data. The first data and the third data have the same data category, and the second data and the fourth data have the same data category. The data categories of the first data and the third data include obstacle categories, and the data categories of the second data and the fourth data include obstacle distances, where the obstacle distances include the distance between the obstacle and the target vehicle. The third processing module is used to determine obstacle information of the target area based on the target data. The obstacle information is used to generate an automatic braking command to control the braking of the target vehicle. The target data includes the first data, the second data, the third data, and the fourth data. The step of determining the obstacle information of the target area based on the target data includes: determining the target obstacle category based on the obstacle category of the target obstacle in the first data, the obstacle category of the target obstacle in the third data, the priority of the first data, and the priority of the third data; Furthermore, the target obstacle distance of the target obstacle is determined based on the obstacle distance of the target obstacle in the second data, the obstacle distance of the target obstacle in the fourth data, the priority of the second data, and the priority of the fourth data.

8. An automatic braking system, characterized in that, include: Data acquisition unit and control unit; The data acquisition unit is used to acquire first images, second images, and radar point cloud data of the target area; The first image is used to characterize the temperature distribution of the target region, and the second image is used to characterize the two-dimensional visual features of the target region; The control unit is connected to the data acquisition unit and is used to determine first data and second data based on the first image, determine third data based on the second image, and determine fourth data based on the radar point cloud data. The first data and the third data have the same data category, and the second data and the fourth data have the same data category. The control unit also determines obstacle information for the target area based on the target data. The obstacle information is used to generate an automatic braking command to control the braking of the target vehicle. The target data includes the first data, the second data, the third data, and the fourth data. The data categories of the first data and the third data include obstacle categories, and the data categories of the second data and the fourth data include obstacle distances. The obstacle distances include the distance between the obstacle and the target vehicle. The step of determining the obstacle information of the target area based on the target data includes: determining the target obstacle category based on the obstacle category of the target obstacle in the first data, the obstacle category of the target obstacle in the third data, the priority of the first data, and the priority of the third data; Furthermore, the target obstacle distance of the target obstacle is determined based on the obstacle distance of the target obstacle in the second data, the obstacle distance of the target obstacle in the fourth data, the priority of the second data, and the priority of the fourth data.

9. A vehicle, characterized in that, The vehicle includes the automatic braking system as described in claim 8.

Images