Obstacle detection method, obstacle detection device and vehicle-mounted system

Abstract

The invention relates to an obstacle detection method, an obstacle detection device and a vehicle-mounted system.The obstacle detection method comprises the steps that echo signals collected by an ultrasonic sensor are obtained, and image data collected by an image sensor are obtained; mapping the echo signal to an aerial view space according to the position distribution information of the ultrasonic sensor to obtain mapping data; extracting a first initial feature from the image data, and mapping the first initial feature to an aerial view space to obtain a first target feature; extracting a second target feature from the mapping data; and performing obstacle detection on the first target feature and the second target feature based on a deep learning algorithm to obtain an obstacle detection result. According to the invention, ultrasonic echo signals and image data are combined, and multi-modal feature acquisition is realized, so that the accuracy of obstacle detection is improved, and the accuracy of collision detection is further improved.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to an obstacle detection method, an obstacle detection device and a vehicle-mounted system. Background Art

[0002] With the development and widespread application of automobile control technology, while it brings convenience to users, it also brings safety hazards. For example, when opening and closing the electric tailgate or reversing the vehicle, there is a possibility of collision with surrounding objects or pedestrians due to visual blind spots or operational errors, causing personal injury or property damage.

[0003] At present, in related technologies, the vehicle trajectory is often calculated based on data such as the vehicle blind spot image, and the trajectory mark is superimposed on the real-time image to form an image for collision detection. By carefully analyzing the interaction between obstacles and trajectory marks in the image, potential collision risks can be revealed. This method relies on the recognition of the edge contour of the obstacle, and there is a risk of misidentification. Therefore, the accuracy of collision detection in related technologies is currently low.

[0004] With regard to the problem of low accuracy of collision detection in related technologies, no effective solution has been proposed so far. Summary of the invention

[0005] In this embodiment, an obstacle detection method, an obstacle detection device, and a vehicle-mounted system are provided to solve the problem of low accuracy of collision detection in the related art.

[0006] In a first aspect, an obstacle detection method is provided in this embodiment, which is used for a vehicle and includes:

[0007] Acquire the echo signal collected by the ultrasonic sensor and acquire the image data collected by the image sensor;

[0008] According to the position distribution information of the ultrasonic sensor, the echo signal is mapped to a bird's-eye view space to obtain mapping data;

[0009] Extracting a first initial feature from the image data and mapping the first initial feature to the bird's-eye view space to obtain a first target feature; and extracting a second target feature from the mapped data;

[0010] Obstacle detection is performed on the first target feature and the second target feature based on a deep learning algorithm to obtain an obstacle detection result.

[0011] In some of the embodiments, mapping the echo signal to a bird's-eye view space according to the position distribution information of the ultrasonic sensor to obtain mapping data includes:

[0012] Acquire a pre-constructed mapping grid structure; the mapping grid structure corresponds to a preset bird's-eye view space and covers a preset obstacle detection area;

[0013] According to the position distribution information of the ultrasonic sensor and the echo signal, the echo estimation value corresponding to each grid unit in the mapping grid structure is calculated to obtain mapping data mapped to the bird's-eye view space.

[0014] In some of the embodiments, calculating the echo estimation value corresponding to each grid unit in the mapping grid structure according to the position distribution information of the ultrasonic sensor and the echo signal to obtain the mapping data mapped to the bird's-eye view space includes:

[0015] Determine, according to different position distribution information of the ultrasonic sensors, the mapping weights corresponding to the respective grid cells of the mapping grid structure of the respective ultrasonic sensors;

[0016] For each grid unit in the mapping grid structure, according to the mapping weights corresponding to the ultrasonic sensors, weighted statistics are performed on the echo signals from different ultrasonic sensors to obtain an echo estimation value corresponding to each grid unit;

[0017] Mapping data mapped to the bird's-eye view space is obtained according to the echo estimation values ​​of all grid cells in the mapping grid structure.

[0018] In some embodiments, performing obstacle detection on the first target feature and the second target feature based on a deep learning algorithm to obtain an obstacle detection result includes:

[0019] According to the trained perception model deployed on the vehicle-mounted chip, obstacle detection is performed on the first target feature and the second target feature to obtain an obstacle detection result; the vehicle-mounted chip runs on the vehicle-mounted system of the vehicle.

[0020] In some embodiments, before performing obstacle detection on the first target feature and the second target feature based on a deep learning algorithm to obtain an obstacle detection result, the method further includes:

[0021] Based on the quantitative perception training, the preset deep learning model is trained to obtain a trained perception model;

[0022] According to a preset model conversion tool, the perception model is converted into a vehicle-mounted file format, and the quantized parameters of the perception model are embedded into a corresponding configuration file;

[0023] The vehicle-mounted file format and the configuration file are compiled into a data format executable by the vehicle-mounted chip, so as to deploy the perception model on the vehicle-mounted chip.

[0024] In some embodiments, before performing obstacle detection on the first target feature and the second target feature based on a deep learning algorithm to obtain an obstacle detection result, the method further includes:

[0025] Obtain preset real vehicle sample data;

[0026] Performing data enhancement on the real vehicle sample data to expand the data to obtain a training sample set;

[0027] Performing two-level target labeling on the training sample set to obtain label data corresponding to the training sample set;

[0028] The preset deep learning model is trained according to the training sample set and the label data to obtain the trained perception model.

[0029] In some embodiments, performing obstacle detection on the first target feature and the second target feature based on a deep learning algorithm to obtain an obstacle detection result includes:

[0030] According to a preset expansion rate corresponding to the second target feature, the second target feature is convolved and then fused with the first target feature to obtain a multimodal feature; the preset expansion rate is determined based on a collection area corresponding to the second target feature; the collection area belongs to a preset obstacle detection area;

[0031] Obstacle detection is performed on the multimodal features based on a deep learning algorithm to obtain an obstacle detection result.

[0032] In some of the embodiments, after obtaining the obstacle detection result, the method further includes:

[0033] If the obstacle detection result indicates that an obstacle exists, determining a position of the obstacle;

[0034] A collision prediction is performed based on the obstacle position and the driving trajectory of the vehicle to obtain a collision prediction result.

[0035] In a second aspect, an obstacle detection device is provided in this embodiment, including: an acquisition module, a mapping module, a feature extraction module and a detection module; wherein:

[0036] The acquisition module is used to acquire the echo signal collected by the ultrasonic sensor and acquire the image data collected by the image sensor;

[0037] The mapping module is used to map the echo signal to a bird's-eye view space according to the position distribution information of the ultrasonic sensor to obtain mapping data;

[0038] The feature extraction module is used to extract a first initial feature from the image data, and map the first initial feature to the bird's-eye view space to obtain a first target feature; and extract a second target feature from the mapped data;

[0039] The detection module is used to perform obstacle detection on the first target feature and the second target feature based on a deep learning algorithm to obtain an obstacle detection result.

[0040] In a third aspect, in this embodiment, a vehicle-mounted system is provided, comprising: an image sensor, an ultrasonic sensor, and a processor; wherein the processor is connected to the image sensor and the ultrasonic sensor;

[0041] The image sensor is used to collect image data within a preset obstacle detection area;

[0042] The ultrasonic sensor is used to collect echo signals within the obstacle detection area;

[0043] The processor is used to execute the obstacle detection method described in the first aspect above.

[0044] Compared with the related art, an obstacle detection method, an obstacle detection device and a vehicle-mounted system are provided in this embodiment. The obstacle detection method obtains the echo signal collected by the ultrasonic sensor and obtains the image data collected by the image sensor; maps the echo signal to the bird's-eye view space according to the position distribution information of the ultrasonic sensor to obtain mapping data; extracts the first initial feature from the image data, and maps the first initial feature to the bird's-eye view space to obtain the first target feature; and extracts the second target feature from the mapping data; performs obstacle detection on the first target feature and the second target feature based on the deep learning algorithm to obtain the obstacle detection result. It combines the ultrasonic echo signal with the image data to realize multimodal feature acquisition, thereby improving the accuracy of obstacle detection, and further improving the accuracy of collision detection.

[0045] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0047] Figure 1 is a hardware structure block diagram of a terminal of the obstacle detection method of this embodiment;

[0048] Figure 2 is a flow chart of the obstacle detection method of this embodiment;

[0049] Figure 3 is a flow chart of an obstacle detection method of some embodiments;

[0050] Figure 4 is a flow chart of a vehicle collision detection method in some embodiments;

[0051] Figure 5 is a schematic diagram of a reversing image display of this embodiment;

[0052] Figure 6 is a structural block diagram of the obstacle detection device of this embodiment;

[0053] Figure 7 is a schematic structural diagram of the vehicle-mounted system of this embodiment;

[0054] Figure 8 This is a vehicle system architecture diagram of this embodiment. DETAILED DESCRIPTION

[0055] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0056] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the", "these" and the like in this application do not represent quantitative restrictions, and they can be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether directly or indirectly. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships. For example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. Usually, the character " / " indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0057] The method embodiment provided in this embodiment can be executed in a terminal, a computer or a similar computing device. For example, running on a terminal, Figure 1 FIG. 1 is a hardware structure diagram of a terminal of the obstacle detection method of this embodiment. Figure 1 As shown, the terminal may include one or more ( Figure 1 Only one is shown in the figure) processor 102 and memory 104 for storing data, wherein processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA. The above terminal may also include a transmission device 106 and an input and output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is for illustration only and does not limit the structure of the above terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0058] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the obstacle detection method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the terminal via a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0059] The transmission device 106 is used to receive or send data via a network. The above network includes a wireless network provided by the communication provider of the terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.

[0060] In this embodiment, an obstacle detection method is provided. Figure 2 is a flow chart of the obstacle detection method of this embodiment. Figure 2 As shown, the process includes the following steps:

[0061] Step S210, acquiring the echo signal collected by the ultrasonic sensor, and acquiring the image data collected by the image sensor.

[0062] Among them, the ultrasonic sensor can be an ultrasonic radar, which can be set on the vehicle or in a preset scene where the vehicle is currently performing obstacle detection. Specifically, the echo signals collected by several ultrasonic sensors distributed in different positions and directions can be obtained. The above-mentioned image sensor can specifically be an RGB camera or other two-dimensional image sensor. The obstacle detection can be applied to the anti-collision detection of the vehicle tailgate, the anti-collision detection of reversing, and the anti-collision detection of other vehicle blind spots. Depending on the application scenario, the number, setting location and method of the above-mentioned ultrasonic sensors and image sensors can be adaptively adjusted.

[0063] Step S220 , mapping the echo signal to the bird's-eye view space according to the position distribution information of the ultrasonic sensor to obtain mapping data.

[0064] Considering that when realizing the fusion of multi-sensor data, in order to integrate and compare the data from different sensors in the same coordinate system, this step maps the echo signals collected by different ultrasonic sensors to the Bird's Eye View (BEV) space, so as to facilitate the integration and alignment of the echo signals of different ultrasonic sensors, thereby improving the accuracy and robustness of the fusion results.

[0065] The bird's-eye view space involved in this step may specifically be a scene view from above the vehicle, which displays obstacles, lane lines and other information around the vehicle in a grid format, so as to facilitate the vehicle-mounted system to make high-precision understanding and planning decisions about the surrounding environment. The position distribution information of the above-mentioned ultrasonic sensors may specifically be the position and direction of different ultrasonic sensors relative to the vehicle coordinate system, as well as the positioning of each ultrasonic sensor relative to the vehicle coordinate system. Based on the above-mentioned position distribution information, the echo signals from different ultrasonic sensors are mapped to the bird's-eye view space to obtain mapping data.

[0066] Step S230, extracting a first initial feature from the image data, and mapping the first initial feature to a bird's-eye view space to obtain a first target feature; and extracting a second target feature from the mapped data.

[0067] Two independent encoders can be used to extract the first initial feature in the image data collected by the image sensor, and to extract the second target feature from the above-mentioned mapping data. The first initial feature can be a perspective feature. After the perspective feature is extracted by the encoder, the perspective feature is converted to the BEV space to obtain the first target feature. Specifically, the perspective feature can be mapped to different scales of the BEV space based on the intrinsic and extrinsic parameters of the image sensor to cover different distance ranges.

[0068] In addition, features such as spatial features, semantic features, voxel features, and attention features can be extracted from the mapping data as second target features. Thus, this step realizes feature extraction of echo signals and feature extraction of image data respectively.

[0069] Step S240: Perform obstacle detection on the first target feature and the second target feature based on a deep learning algorithm to obtain an obstacle detection result.

[0070] The first target feature and the second target feature can be combined into a multimodal feature, and the multimodal feature can be processed using a trained BEV perception model to predict the category and position of the target obstacle in a preset obstacle detection area, and then the category and position of the target obstacle are output as the obstacle detection result.

[0071] In related technologies, the vehicle trajectory is often calculated based on data such as the vehicle blind spot image, and then the trajectory mark is superimposed on the real-time image to analyze the interaction between the obstacle and the trajectory mark in the image to perform collision detection. This process relies on image recognition of the edge contour of the obstacle, which has a high risk of misidentification, resulting in low accuracy of collision detection.

[0072] Different from this, this embodiment uses the image sensor to collect image data through the above steps S210 to S240, applies deep learning technology, performs real-time detection and analysis of potential obstacles, and combines the beam characteristics of the ultrasonic sensor to achieve accurate collision prevention. On the one hand, the echo signal collected by the ultrasonic sensor introduced in this embodiment can compensate for the performance loss of the image sensor in low visibility, weak light and bad weather conditions to a certain extent when processed by the deep learning algorithm. On the other hand, ultrasonic sensors such as ultrasonic radars have low costs and high cost performance. In addition, image sensors and software algorithms are easy to integrate with existing automotive electronic systems, so the labor cost of hardware modification is also low. The deep learning algorithm used can realize accurate identification of multiple types of obstacles and has strong scalability. Therefore, this embodiment can also reduce hardware costs while ensuring detection accuracy, and has good potential for expansion and promotion of application.

[0073] In the above steps S210 to S240, the echo signal collected by the ultrasonic sensor is obtained, and the image data collected by the image sensor is obtained; the echo signal is mapped to the bird's-eye view space according to the position distribution information of the ultrasonic sensor to obtain the mapping data; the first initial feature is extracted from the image data, and the first initial feature is mapped to the bird's-eye view space to obtain the first target feature; and the second target feature is extracted from the mapping data; obstacle detection is performed on the first target feature and the second target feature based on the deep learning algorithm to obtain the obstacle detection result. It combines the ultrasonic echo signal with the image data to realize multimodal feature acquisition, thereby improving the accuracy of obstacle detection, and further improving the accuracy of collision detection.

[0074] In one embodiment, based on the above step S220, according to the location distribution information of the ultrasonic sensor, the echo signal is mapped to the bird's-eye view space to obtain mapping data, which may specifically include:

[0075] A pre-constructed mapping grid structure is obtained; the mapping grid structure corresponds to a preset bird's-eye view space and covers a preset obstacle detection area; according to the position distribution information and echo signal of the ultrasonic sensor, the echo estimation value corresponding to each grid unit in the mapping grid structure is calculated to obtain mapping data mapped to the bird's-eye view space.

[0076] Among them, a grid structure centered on the image sensor position, that is, the camera position, can be pre-constructed to map the BEV space. The grid size and resolution in the grid structure can be pre-set according to the area where obstacle detection is required. Specifically, the grid structure needs to fully cover the preset obstacle detection area and be able to achieve the preset accuracy requirements in the actual application scenario in terms of detection accuracy to capture the required details. The construction and setting of the specific grid structure can be set according to the actual application scenario, and this embodiment does not specifically limit this.

[0077] The location distribution information of the ultrasonic sensor can be obtained by analyzing the installation parameters and calibration data of the ultrasonic sensor to obtain the position, direction and positioning of the ultrasonic sensor relative to the vehicle coordinate system. In addition, for each grid unit, the distance and angle between the grid unit and each ultrasonic sensor can be calculated based on the location distribution information of the ultrasonic sensor and the specific coordinates of the grid unit. By calculating the distance and angle between each grid unit and each ultrasonic sensor, the exact position of the obstacle relative to the ultrasonic sensor and the vehicle coordinate system can be determined.

[0078] Afterwards, for each grid unit in the grid structure, the echo estimation value corresponding to each grid unit is calculated according to the position distribution information of the ultrasonic sensor. The echo estimation value may specifically be an echo amplitude value. In addition, in view of the fact that the measurement data provided by the ultrasonic sensor is discrete and the grid unit represents a continuous space, the amplitude value of the echo signal of the ultrasonic sensor may be interpolated to finally obtain the echo amplitude value of each grid unit.

[0079] In this embodiment, by introducing grid cells, a unified reference frame is provided, so that data from different ultrasonic sensors can be integrated and compared in the same coordinate system, thereby improving the accuracy and robustness of multi-sensor data fusion.

[0080] In particular, in one embodiment, according to the location distribution information and the echo signal of the ultrasonic sensor, the echo estimation value corresponding to each grid unit in the mapping grid structure is calculated to obtain the mapping data mapped to the bird's-eye view space, which may include:

[0081] According to the position distribution information of different ultrasonic sensors, the mapping weights corresponding to each ultrasonic sensor in each grid unit of the mapping grid structure are determined; for each grid unit in the mapping grid structure, the echo signals from different ultrasonic sensors are weighted statistically analyzed according to the mapping weights corresponding to each ultrasonic sensor, so as to obtain the echo estimation value corresponding to each grid unit; according to the echo estimation values ​​of all grid units in the mapping grid structure, the mapping data mapped to the bird's-eye view space is obtained.

[0082] In this embodiment, considering that multiple ultrasonic sensors are deployed, the echo signals from different ultrasonic sensors in each grid unit can be weighted and summed. For each grid unit, the mapping weight of each ultrasonic sensor in the grid unit is determined based on the distance, angle and signal-to-noise ratio of each ultrasonic sensor relative to the grid unit, and then in the grid unit, the echo signals collected by each ultrasonic sensor are weighted and counted according to the mapping weights corresponding to each ultrasonic sensor, and the echo estimation value corresponding to the grid unit is obtained. Finally, after obtaining the echo estimation values ​​corresponding to all grid units, the mapping data of the BEV space to be set is obtained.

[0083] In this embodiment, by assigning corresponding mapping weights based on the position distribution information of different ultrasonic sensors relative to each grid unit, and then performing weighted statistics on the echo signals from different ultrasonic sensors based on the mapping weights, the accuracy and consistency of multi-sensor data fusion can be improved, and the obstacle detection's perception of the environment can be enhanced. Through this fusion process, the advantages of each ultrasonic sensor can be fully utilized to improve the accuracy and reliability of information.

[0084] Additionally, in one embodiment, based on the above step S240, obstacle detection is performed on the first target feature and the second target feature based on a deep learning algorithm to obtain an obstacle detection result, which may specifically include:

[0085] According to the trained perception model deployed on the vehicle-mounted chip, obstacle detection is performed on the first target feature and the second target feature to obtain an obstacle detection result; the vehicle-mounted chip runs on the vehicle-mounted system. In this embodiment, obstacle detection can be performed based on a deep learning model suitable for the vehicle-mounted environment to obtain an obstacle detection result. The above-mentioned vehicle-mounted chip can specifically be a Qualcomm vehicle-mounted chip.

[0086] In one embodiment, before performing obstacle detection on the first target feature and the second target feature based on the deep learning algorithm to obtain the obstacle detection result, the obstacle detection method may further include:

[0087] Based on quantized perception training, the preset deep learning model is trained to obtain a trained perception model; according to the preset model conversion tool, the perception model is converted into a vehicle-mounted file format, and the quantization parameters of the perception model are embedded into the corresponding configuration file; the vehicle-mounted file format and the configuration file are compiled into a data format executable by the vehicle-mounted chip to deploy the perception model on the vehicle-mounted chip.

[0088] First, in the model training stage, Quantization-Aware Training (QAT) can be used to accurately capture model parameters and convert model parameters from floating-point representation to the quantization parameters required for low-precision integers. After the model training is completed, the trained perception model can be converted into a specific file format suitable for the vehicle environment using the Qualcomm Neural Network (QNN) model conversion tool. At the same time, the corresponding quantization parameters are embedded in the JSON configuration file of the converted perception model, which is then compiled into a binary format executable by QNN to facilitate efficient reasoning on a neural processing unit (NPU) supported by the software development kit (SDK) of the Qualcomm neural network processing engine. It can be understood that the above-mentioned perception model can be implemented based on any deep learning model suitable for target detection, and this embodiment does not specifically limit this.

[0089] Therefore, this embodiment can efficiently deploy the deep learning model to the vehicle-mounted chip, improving the performance of the deep learning algorithm on the vehicle-mounted chip. Through this model compression technology, the deep learning algorithm can be efficiently run on the limited vehicle-side computing resources. Compared with the related technologies, the high computing requirements of image processing and trajectory planning make it difficult for vehicle-mounted hardware equipment to meet the requirements. The demand for precise vehicle structural parameters and real-time data synchronization also increases the complexity of hardware design and debugging. This embodiment can make full use of the hardware characteristics of the vehicle-mounted chip, reduce the consumption of computing resources of the deep learning algorithm on the vehicle-mounted hardware equipment during driving, and solve the problems of limited computing resources and real-time performance.

[0090] Optionally, in one embodiment, before performing obstacle detection on the first target feature and the second target feature based on the deep learning algorithm to obtain the obstacle detection result, the obstacle detection method may further include:

[0091] Obtain preset real vehicle sample data; perform data enhancement on the real vehicle sample data to expand it to obtain a training sample set; perform two-level target annotation on the training sample set to obtain label data corresponding to the training sample set; train the preset deep learning model based on the training sample set and the label data to obtain a trained perception model.

[0092] In the model training stage, this embodiment collects more real vehicle data as real vehicle sample data. According to the actual application scenario, the specific detection target category can be divided into a number of primary targets and a corresponding number of secondary targets. For example, the specific division method of the two-level detection target category can be as follows:

[0093]

[0094] In addition, during the data collection process, in order to enable the deep learning model to adapt to different environments and lighting conditions, the training sample set can be expanded through real shooting and data enhancement. Specific data enhancement methods include but are not limited to one or more of the following: random rotation, scaling, translation, flipping, color jittering to adjust brightness, contrast and saturation, and performing affine transformation and perspective transformation while introducing Gaussian noise. By performing the above-mentioned data enhancement processing on the collected data, the deep learning model's target recognition ability for different viewing angles, lighting conditions and occlusion conditions can be improved. By training the model based on the annotation of secondary detection targets, more detailed obstacle classification can be achieved, further reducing the risk of misidentification of obstacles, improving the accuracy and adaptability of obstacle detection in different application scenarios, and ultimately improving the accuracy and adaptability of collision detection.

[0095] In particular, in one embodiment, based on the above step S240, obstacle detection is performed on the first target feature and the second target feature based on a deep learning algorithm to obtain an obstacle detection result, which may also include:

[0096] According to the preset expansion rate corresponding to the second target feature, the second target feature is convolved and then fused with the first target feature to obtain a multimodal feature; the preset expansion rate is determined based on the collection area corresponding to the second target feature; the collection area belongs to the preset obstacle detection area; obstacle detection is performed on the multimodal feature based on the deep learning algorithm to obtain an obstacle detection result.

[0097] In this embodiment, in order to solve the misalignment problem between the data collected by the ultrasonic sensor and the data collected by the image sensor, the second target feature corresponding to the ultrasonic sensor is content-aware. For the second target feature, the predetermined area related to vehicle driving safety (for example, the area where there are target objects such as human bodies and vehicles) is taken as the important area, and the area far away from the vehicle or with less impact on driving decisions is taken as the secondary area. A smaller expansion rate is assigned to the important area to capture finer features, and a larger expansion rate is assigned to the secondary area. The second target feature is convolved based on the expansion rate. So that the processed second target feature and the first target feature can achieve data alignment, perform feature map fusion, and generate a multimodal feature map as the above-mentioned multimodal feature.

[0098] In one embodiment, after obtaining the above obstacle detection result, the above obstacle detection method may further include:

[0099] When the obstacle detection result indicates that there is an obstacle, the obstacle position is determined; a collision prediction is performed based on the obstacle position and the vehicle's driving trajectory to obtain a collision prediction result.

[0100] That is, when a target obstacle is detected based on the above multimodal features, a collision prediction is performed based on the position of the obstacle and the vehicle trajectory to assess the potential collision risk. Based on this, the present embodiment can provide accurate collision warning information, thereby assisting the driver to complete the reversing operation safely and efficiently.

[0101] Figure 3 is a flow chart of an obstacle detection method of some embodiments, such as Figure 3 As shown, the obstacle detection method may include the following steps:

[0102] Step S301: collecting image data through an image sensor.

[0103] Step S302, collecting echo signals through an ultrasonic sensor. Step S302 and step S301 may be performed simultaneously or sequentially, and the order of execution is not limited.

[0104] Step S303, inputting the image data obtained in step S301 into a feature extraction network to obtain a first initial feature.

[0105] Step S304, preprocessing the echo signal to obtain an echo signal mapped to the BEV space. The execution order of step S304 and step S303 is not limited.

[0106] Step S305: Perform BEV space conversion on the first initial feature to obtain a first target feature.

[0107] Step S306, input the data processed in step S304 into a feature extraction network to obtain a second target feature. The feature extraction networks in step S303 and step S306 may be two independent feature extraction networks.

[0108] Step S307: perform feature fusion on the first target feature and the second target feature to obtain a multimodal feature.

[0109] Step S308: input the multimodal features into the trained BEV perception model to perform obstacle detection and obtain obstacle detection results.

[0110] Step S309: output the obstacle category and obstacle position in the obstacle detection result.

[0111] In the above steps S301 to S309, real-time image data is captured by the image sensor installed at the rear of the vehicle, and obstacle features are extracted through a neural network in combination with the data of the ultrasonic sensor. Then, the features are analyzed using a deep learning algorithm to identify and classify obstacles in the image. The specific information of the obstacles can be accurately parsed, thereby realizing a multi-modal end-to-end obstacle detection and improving the accuracy of obstacle detection.

[0112] Figure 4 is a flow chart of a vehicle collision detection method in some embodiments, such as Figure 4 As shown, the vehicle collision detection method includes the following steps:

[0113] Step S401, acquiring real-time image data collected by an image sensor behind the vehicle and an echo signal collected by an ultrasonic sensor.

[0114] Step S402, perform obstacle detection on the data obtained in step S401 according to the trained perception model to obtain an obstacle detection result. The specific obstacle detection method can refer to any obstacle detection method provided in the above embodiments, and this embodiment does not specifically limit this.

[0115] Step S403, judging whether the obstacle confidence is greater than a preset confidence threshold according to the obstacle detection result; if so, executing step S404; otherwise, executing step S407.

[0116] Step S404: determine whether there is an obstacle, and determine the obstacle frame according to the obstacle detection result.

[0117] Step S405, determining whether the distance between the obstacle frame and the vehicle reversing trajectory mark is less than a preset distance threshold; if so, executing step S406; otherwise, executing step S408.

[0118] Step S406: issuing a collision warning prompt.

[0119] Step S407: determine that there is no obstacle and do not issue a warning.

[0120] Step S408: Determine that the obstacle is at a safe distance and do not issue a warning.

[0121] In the above steps S401 to S408, the image sensor at the rear of the vehicle is used to capture real-time reversing images, while the ultrasonic sensor synchronously collects echo signals from the surrounding environment. These data are then input into a trained end-to-end deep learning model for in-depth analysis. The model can accurately parse the specific information of obstacles and evaluate the potential collision risk in combination with the expected trajectory identification line when reversing. Based on this evaluation, the system will eventually provide accurate warning information to assist the driver to complete the reversing operation safely and efficiently.

[0122] Figure 5 FIG. 1 is a schematic diagram of a reversing image display of this embodiment. Figure 5 As shown, the left image is the BEV view during the reversing process after BEV spatial mapping; the right image is the reversing image captured by the image sensor at the rear of the vehicle.

[0123] In this embodiment, an obstacle detection device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. The terms "module", "unit", "subunit", etc. used below can be a combination of software and / or hardware that implements the predetermined functions. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0124] Figure 6 is a structural block diagram of the obstacle detection device 60 of this embodiment, as shown in Figure 6 As shown, the obstacle detection device 60 includes: an acquisition module 62, a mapping module 64, a feature extraction module 66 and a detection module 68; wherein:

[0125] An acquisition module 62 is used to acquire the echo signal collected by the ultrasonic sensor and to acquire the image data collected by the image sensor; a mapping module 64 is used to map the echo signal to the bird's-eye view space according to the position distribution information of the ultrasonic sensor to obtain mapping data; a feature extraction module 66 is used to extract a first initial feature from the image data, and map the first initial feature to the bird's-eye view space to obtain a first target feature; and, to extract a second target feature from the mapping data; a detection module 68 is used to perform obstacle detection on the first target feature and the second target feature based on a deep learning algorithm to obtain an obstacle detection result.

[0126] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0127] In this embodiment, a vehicle-mounted system is also provided. Figure 7 Schematic diagram of the structure of the vehicle-mounted system 70 of this embodiment. Figure 7 As shown, the vehicle-mounted system 70 may include: an image sensor 72, an ultrasonic sensor 74, and a processor 76; wherein the processor 76 is connected to the image sensor 72 and the ultrasonic sensor 74;

[0128] The image sensor 72 is used to collect image data within a preset obstacle detection area; the ultrasonic sensor 74 is used to collect echo signals within the obstacle detection area; and the processor 76 is used to execute the obstacle detection method provided in any of the above embodiments.

[0129] The image sensor 72 may be a vehicle-mounted camera for capturing image data of the environment around the vehicle. The vehicle may be equipped with several ultrasonic sensors for detecting obstacles near the vehicle and returning signals. A parking assist system (PAS) may be provided in the processor to receive image data and echo signals, help the driver detect obstacles around the vehicle, and automatically control the vehicle.

[0130] Optionally, Figure 8 is a vehicle system architecture diagram of this embodiment, such as Figure 8 As shown, the vehicle system includes a vehicle camera, an ultrasonic sensor, a flat panel display (Flat Panel Display, referred to as FPD), a parking assistance system (PAS), a Gigabit Multimedia Serial Link (GMSL), a rear view camera control unit (Rear View Mirror Control, referred to as RVMC), a driving information and entertainment host (Display Head Unit, referred to as DHU), a distributed bus system (Flexray), a body domain controller (Central Electronic Module, referred to as CEM), a controller area network (Controller Area Network, referred to as CAN) and a power tailgate module (Power Operated Tailgate, referred to as POT).

[0131] Among them, the on-board camera captures the environment around the vehicle. The on-board camera transmits images through the communication interface connected to the flat-panel display in the vehicle. The ultrasonic sensor equipped in the vehicle is used to detect obstacles near the vehicle and return signals. The parking assistance system receives the video image transmitted by the camera and the echo signal collected by the ultrasonic sensor to help the driver detect surrounding obstacles and automatically control the vehicle when necessary. The multimedia devices in the vehicle are connected through the high-speed serial communication protocol of the gigabit multimedia serial link. The rearview camera control unit is used to capture the surrounding environment behind the vehicle. The vehicle rearview camera and the on-board host are connected through the high-speed serial communication technology of low voltage differential signaling (LVDS). The driving information and entertainment host is used to process data from vehicle sensors. The visual algorithm and collision detection system are deployed on the host, and finally the collision detection signal is obtained. The signal is transmitted to the key safety power control system through the on-board communication protocol known for its high reliability and real-time performance. The body domain controller will receive the signal of the DHU. Among them, in the tailgate anti-collision detection, the body domain controller and the electric tailgate module are connected through the controller area network, and finally the electric tailgate will control the opening or stopping of the tailgate. For example, the number of vehicle-mounted cameras may be 4, and the number of ultrasonic sensors may be 12. It is understood that the architecture and the number of components of the vehicle-mounted system herein are only examples and do not constitute a specific limitation on the architecture of the vehicle-mounted system.

[0132] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and will not be repeated in this embodiment.

[0133] It should be understood that the specific embodiments described herein are only used to explain the application, rather than to limit it. Based on the embodiments provided in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of this application.

[0134] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0135] Obviously, the drawings are only some examples or embodiments of the present application. For ordinary technicians in the field, the present application can also be applied to other similar situations based on these drawings without creative work. In addition, it is understandable that although the work done in this development process may be complicated and lengthy, for ordinary technicians in the field, certain changes in design, manufacturing or production based on the technical content disclosed in this application are only conventional technical means and should not be regarded as insufficient content disclosed in this application.

[0136] The term "embodiment" in this application refers to a specific feature, structure or characteristic described in conjunction with the embodiment that can be included in at least one embodiment of the present application. The appearance of this phrase in various locations in the specification does not necessarily mean the same embodiment, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. It is clearly or implicitly understood by those of ordinary skill in the art that the embodiments described in this application can be combined with other embodiments without conflict.

[0137] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of patent protection. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the attached claims.

Claims

1. An obstacle detection method for a vehicle, characterized in that: include: Acquire the echo signal collected by the ultrasonic sensor and acquire the image data collected by the image sensor; According to the position distribution information of the ultrasonic sensor, the echo signal is mapped to a bird's-eye view space to obtain mapping data; Extracting a first initial feature from the image data, and mapping the first initial feature to the bird's-eye view space to obtain a first target feature; and, extracting a second target feature from the mapping data; Obstacle detection is performed on the first target feature and the second target feature based on a deep learning algorithm to obtain an obstacle detection result.

2. The obstacle detection method according to claim 1, characterized in that: According to the position distribution information of the ultrasonic sensor, the echo signal is mapped to the bird's-eye view space to obtain mapping data, including: Acquire a pre-constructed mapping grid structure; the mapping grid structure corresponds to a preset bird's-eye view space and covers a preset obstacle detection area; According to the position distribution information of the ultrasonic sensor and the echo signal, the echo estimation value corresponding to each grid unit in the mapping grid structure is calculated to obtain mapping data mapped to the bird's-eye view space.

3. The obstacle detection method according to claim 2, characterized in that: According to the position distribution information of the ultrasonic sensor and the echo signal, the echo estimation value corresponding to each grid unit in the mapping grid structure is calculated to obtain mapping data mapped to the bird's-eye view space, including: Determine, according to different position distribution information of the ultrasonic sensors, the mapping weights corresponding to the respective grid cells of the mapping grid structure of the respective ultrasonic sensors; For each grid unit in the mapping grid structure, according to the mapping weights corresponding to the ultrasonic sensors, weighted statistics are performed on the echo signals from different ultrasonic sensors to obtain an echo estimation value corresponding to each grid unit; Mapping data mapped to the bird's-eye view space is obtained according to the echo estimation values ​​of all grid cells in the mapping grid structure.

4. The obstacle detection method according to claim 1, characterized in that: Performing obstacle detection on the first target feature and the second target feature based on a deep learning algorithm to obtain an obstacle detection result includes: According to the trained perception model deployed on the vehicle-mounted chip, obstacle detection is performed on the first target feature and the second target feature to obtain an obstacle detection result; the vehicle-mounted chip runs on the vehicle-mounted system of the vehicle.

5. The obstacle detection method according to claim 4, characterized in that: Before performing obstacle detection on the first target feature and the second target feature based on the deep learning algorithm to obtain an obstacle detection result, the method further includes: Based on the quantitative perception training, the preset deep learning model is trained to obtain a trained perception model; According to a preset model conversion tool, the perception model is converted into a vehicle-mounted file format, and the quantized parameters of the perception model are embedded into a corresponding configuration file; The vehicle-mounted file format and the configuration file are compiled into a data format executable by the vehicle-mounted chip, so as to deploy the perception model on the vehicle-mounted chip.

6. The obstacle detection method according to claim 4, characterized in that: Before performing obstacle detection on the first target feature and the second target feature based on the deep learning algorithm to obtain an obstacle detection result, the method further includes: Obtain preset real vehicle sample data; Performing data enhancement on the real vehicle sample data to expand the data to obtain a training sample set; Performing two-level target labeling on the training sample set to obtain label data corresponding to the training sample set; The preset deep learning model is trained according to the training sample set and the label data to obtain the trained perception model.

7. The obstacle detection method according to claim 1, characterized in that: Performing obstacle detection on the first target feature and the second target feature based on a deep learning algorithm to obtain an obstacle detection result includes: According to a preset expansion rate corresponding to the second target feature, the second target feature is convolved and then fused with the first target feature to obtain a multimodal feature; the preset expansion rate is determined based on a collection area corresponding to the second target feature; the collection area belongs to a preset obstacle detection area; Obstacle detection is performed on the multimodal features based on a deep learning algorithm to obtain an obstacle detection result.

8. The obstacle detection method according to any one of claims 1 to 7, characterized in that: After obtaining the obstacle detection result, the method further includes: If the obstacle detection result indicates that an obstacle exists, determining a position of the obstacle; A collision prediction is performed based on the obstacle position and the driving trajectory of the vehicle to obtain a collision prediction result.

9. An obstacle detection device, characterized in that: include: Acquisition module, mapping module, feature extraction module and detection module; wherein: The acquisition module is used to acquire the echo signal collected by the ultrasonic sensor and acquire the image data collected by the image sensor; The mapping module is used to map the echo signal to a bird's-eye view space according to the position distribution information of the ultrasonic sensor to obtain mapping data; The feature extraction module is used to extract a first initial feature from the image data, and map the first initial feature to the bird's-eye view space to obtain a first target feature; and extract a second target feature from the mapped data; The detection module is used to perform obstacle detection on the first target feature and the second target feature based on a deep learning algorithm to obtain an obstacle detection result.

10. A vehicle-mounted system, characterized in that: include: An image sensor, an ultrasonic sensor and a processor; wherein the processor is connected to the image sensor and the ultrasonic sensor; The image sensor is used to collect image data within a preset obstacle detection area; The ultrasonic sensor is used to collect echo signals within the obstacle detection area; The processor is used to execute the obstacle detection method according to any one of claims 1 to 8.

Images