Vehicle obstacle avoidance method and device and storage medium

Abstract

The invention provides a vehicle obstacle avoidance method and device and a storage medium, and belongs to the technical field of vehicles. The method comprises the steps that image recognition is conducted on a plurality of thermal images collected by a thermal infrared imager, thermal imaging information of each thermal image is obtained, and the thermal imaging information comprises at least one of the category and the size of a target object in the thermal image; determining movement information of the target object based on the thermal imaging information of the plurality of thermal images; based on the ultrasonic radar, a first distance is determined, and the first distance is the distance between the target object and the vehicle; and determining a danger level of the target object based on the thermal imaging information, the movement information and the first distance of the target object, and executing an obstacle avoidance strategy corresponding to the danger level. According to the technical scheme, the obstacles around the vehicle and the danger level of the obstacles can be accurately detected, and then the corresponding obstacle avoidance decision is executed, so that collision between the vehicle and the obstacles is avoided, and the driving safety is improved.

Description

Technical Field

[0001] This application relates to the technical field of vehicles, and particularly to a vehicle obstacle avoidance method, device, and storage medium. Background Art

[0002] When driving in a narrow parking space, low-light environment, or complex light environment, such as when reversing in the above-mentioned environments, drivers usually have certain blind spots in their vision, and there may be children, animals, or other moving obstacles in the blind spots, increasing the risk of vehicle collisions. Therefore, how to accurately identify and locate moving obstacles around the vehicle and avoid vehicle collisions with obstacles to improve driving safety is a technical problem that needs to be solved. Summary of the Invention

[0003] Embodiments of this application provide a vehicle obstacle avoidance method, device, and storage medium, which can improve driving safety. The technical solutions are as follows:

[0004] On the one hand, a vehicle obstacle avoidance method is provided, which is applied to a vehicle. An infrared thermal imager and an ultrasonic radar are installed on the vehicle. The method includes:

[0005] Performing image recognition on multiple thermal images collected by the infrared thermal imager to obtain thermal imaging information of each thermal image, where the thermal imaging information includes at least one of the category and size of the target object in the thermal image;

[0006] Based on the thermal imaging information of the multiple thermal images, determining the movement information of the target object, where the movement information includes the movement direction and movement speed;

[0007] Based on the ultrasonic radar, determining a first distance, where the first distance is the distance between the target object and the vehicle;

[0008] Based on the thermal imaging information, movement information of the target object, and the first distance, determining the danger level of the target object, and executing an obstacle avoidance strategy corresponding to the danger level.

[0009] On the other hand, a vehicle obstacle avoidance device is provided, which is configured in a vehicle. An infrared thermal imager and an ultrasonic radar are installed on the vehicle. The device includes:

[0010] An identification module, configured to perform image recognition on multiple thermal images collected by the infrared thermal imager to obtain thermal imaging information of each thermal image, where the thermal imaging information includes at least one of the category and size of the target object in the thermal image;

[0011] A determination module, configured to determine the movement information of the target object based on the thermal imaging information of the multiple thermal images, where the movement information includes a movement direction and a transfer speed;

[0012] The determination module is further configured to determine a first distance based on the ultrasonic radar, where the first distance is the distance between the target object and the vehicle;

[0013] An execution module, configured to determine the danger level of the target object based on the thermal imaging information, movement information of the target object, and the first distance, and execute an obstacle avoidance strategy corresponding to the danger level.

[0014] In some embodiments, the determination module is configured to determine the movement direction and movement distance of a target pixel point based on the thermal imaging information of the multiple thermal images, where the target pixel point is the pixel point in the thermal image representing the target object; and determine the movement information of the target object based on the movement direction and movement distance of the target pixel point.

[0015] In some embodiments, the determination module is configured to determine the optical flow vector of a target pixel point based on the thermal imaging information of the multiple thermal images, where the target pixel point is the pixel point in the thermal image representing the target object; and determine the movement information of the target object based on the magnitude and direction of the optical flow vector.

[0016] In some embodiments, the device further includes:

[0017] An adjustment module, configured to predict the movement trajectory of the target object based on the movement information of the target object, where the movement trajectory represents multiple positions of the target object in a future time period; in the case where the movement trajectory of the target object exceeds the field of view range of the infrared thermal imager, determine the angle adjustment parameter of the infrared thermal imager based on the movement information and the movement trajectory of the target object; and adjust the angle of the infrared thermal imager based on the angle adjustment parameter so that the field of view range of the infrared thermal imager automatically tracks the target object.

[0018] In some embodiments, the adjustment module is configured to determine the relative speed between the target object and the vehicle based on the movement information of the target object and the driving speed of the vehicle; and determine the angle adjustment parameter of the infrared thermal imager based on the movement information, the movement trajectory, the relative speed of the target object, and the distance between the target object and the vehicle.

[0019] In some embodiments, two infrared thermal imagers are installed on the vehicle. The determining module is further configured to determine the parallax value of the target object at the moment based on two thermal images collected by the two infrared thermal imagers at the same moment, where the parallax value represents the relative displacement of the heat source center of the target object in the two thermal images; determine a second distance based on the baseline distance, the parallax value, and the focal length of the infrared thermal imager, where the baseline distance represents the distance between the two infrared thermal imagers, and the second distance is the distance between the target object and the vehicle at the moment; perform weighted summation on the first distance and the second distance to obtain the distance between the target object and the vehicle.

[0020] In some embodiments, the execution module is configured to determine the danger level of the target object based on the thermal imaging information, movement information of the target object, and the distance between the target object and the vehicle. The danger level is positively correlated with the moving speed of the target object and negatively correlated with the distance between the target object and the vehicle.

[0021] In some embodiments, the danger level includes a low level, a medium level, and a high level; when the driving mode of the vehicle is the autonomous driving mode, the execution module is configured to update the driving parameters of the vehicle, where the driving parameters include at least one of the driving speed and the driving path, when the danger level of the target object is the low level; output a prompt message through the in-vehicle large screen and reduce the driving speed of the vehicle when the danger level of the target object is the medium level; output a prompt message through the in-vehicle large screen and trigger the emergency braking of the vehicle when the danger level of the target object is the high level.

[0022] In some embodiments, the danger level includes a low level, a medium level, and a high level; when the driving mode of the vehicle is the manual driving mode, the execution module is configured to display a prompt message through the in-vehicle large screen, where the prompt message indicates that a target object is detected around the vehicle, when the danger level of the target object is the low level; display and broadcast the prompt message through the in-vehicle large screen when the danger level of the target object is the medium level; display and broadcast the prompt message through the in-vehicle large screen and trigger the emergency braking of the vehicle when the danger level of the target object is the high level.

[0023] On the other hand, a computer-readable storage medium is provided, in which at least one segment of computer program is stored. The at least one segment of computer program is loaded and executed by a processor to implement the vehicle obstacle avoidance method in the embodiments of the present application.

[0024] On the other hand, a computer program product is provided, including a computer program which is executed by a processor to implement the vehicle obstacle avoidance method in the embodiments of the present application.

[0025] The embodiments of the present application provide a vehicle obstacle avoidance method, which can accurately and timely detect obstacles existing around the vehicle and the danger levels of the obstacles according to the thermal images collected by an infrared thermal imager and the distance information detected by an ultrasonic radar during driving, and then execute corresponding obstacle avoidance decisions to avoid collisions between the vehicle and the obstacles, improving driving safety. In addition, the above method adopts a multi-sensor fusion strategy, overcoming the limitations of a single sensor and further improving the accuracy of obstacle detection and vehicle obstacle avoidance. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] To more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0027] Figure 1 is a schematic diagram of the implementation environment of a vehicle obstacle avoidance method provided by an embodiment of the present application;

[0028] Figure 2 is a flowchart of a vehicle obstacle avoidance method provided by an embodiment of the present application;

[0029] Figure 3 is a flowchart of another vehicle obstacle avoidance method provided by an embodiment of the present application;

[0030] Figure 4 is a flowchart of the angle adjustment of an infrared thermal imager provided by an embodiment of the present application;

[0031] Figure 5 is a flowchart of executing an obstacle avoidance strategy provided by an embodiment of the present application;

[0032] Figure 6 is a block diagram of a vehicle obstacle avoidance device provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0033] To make the objectives, technical solutions, and advantages of the present application clearer, the following will further describe the embodiments of the present application in detail with reference to the drawings.

[0034] In this application, terms such as "first" and "second" are used to distinguish between identical or similar items with basically the same functions. It should be understood that there is no logical or chronological dependence between "first", "second", and "nth", nor are the quantity and execution order limited.

[0035] In this application, the term "at least one" means one or more, and the meaning of "multiple" means two or more.

[0036] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data for analysis, stored data, displayed data, etc.), and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data need to comply with the relevant laws, regulations, and standards of relevant countries and regions.

[0037] Figure 1 is a schematic diagram of the implementation environment of a vehicle obstacle avoidance method provided according to an embodiment of this application. Refer to Figure 1 This implementation environment includes: vehicle 101, infrared thermal imager 102, and ultrasonic radar 103.

[0038] Among them, an infrared thermal imager 102 and an ultrasonic radar 103 are installed on the vehicle 101. For example, the infrared thermal imager 102 and the ultrasonic radar can be installed at positions such as the rear bumper, front bumper, and sides near the front and rear wheels of the vehicle 101. The embodiments of this application do not limit the quantity and installation positions of the infrared thermal imager 102 and the ultrasonic radar 103.

[0039] Optionally, the infrared thermal imager can be installed on the vehicle 101 through a pan-tilt head 104. Furthermore, by adjusting the rotation angle of the pan-tilt head 104, the field of view of the infrared thermal imager 102 can be conveniently adjusted, enabling the infrared thermal imager 102 to automatically track moving obstacles.

[0040] The infrared thermal imager 102 can be a cooled infrared thermal imager with a short exposure time and a high frame rate. Among them, the detector of the cooled infrared thermal imager works in a low-temperature environment, which can reduce thermal noise, thereby improving the clarity and resolution of the thermal image and being suitable for low-light or harsh weather environments. The short exposure time and high frame rate can significantly improve the capture ability of the infrared thermal imager for fast-moving objects. Among them, the short exposure time reduces motion blur, enabling the infrared thermal imager to capture clear and stable thermal images even when the object is moving at high speed. The high frame rate can provide a relatively smooth thermal image output, which is helpful for real-time tracking and dynamic analysis of objects in the scene.

[0041] The ultrasonic radar 103 can be an ultrasonic sensor for detecting distance information. The ultrasonic radar 103 is capable of emitting ultrasonic waves. When the ultrasonic waves encounter an obstacle, they will be reflected back. The ultrasonic radar 103 receives the reflected waves and calculates the distance between the vehicle 101 and the obstacle based on the time difference between transmission and reception, in combination with the propagation speed of ultrasonic waves in the air.

[0042] For example, when the infrared thermal imager 102 and the ultrasonic radar 103 are installed on the rear bumper of the vehicle 101, during reverse parking, the type and size of the rear obstacle can be detected by the infrared thermal imager 102, and the distance between the rear obstacle and the vehicle 101 can be detected by the ultrasonic radar. Thus, the reverse parking environment of the vehicle can be detected more accurately and comprehensively to assist the vehicle 101 in safe reverse parking. Another example is in the automatic parking scenario. The infrared thermal imager 102 and the ultrasonic radar 103 installed on the front and rear bumpers and the sides of the vehicle work together to jointly detect the obstacles around the vehicle, thereby assisting the vehicle in automatic parking and improving the driving safety.

[0043] Figure 2 It is a flowchart of a vehicle obstacle avoidance method provided according to an embodiment of the present application. As Figure 2 shown, taking the execution of this method by the vehicle as an example, this method includes the following steps:

[0044] 201. The vehicle performs image recognition on multiple thermal images collected by the infrared thermal imager to obtain the thermal imaging information of each thermal image. The thermal imaging information includes at least one of the type and size of the target object in the thermal image.

[0045] In the embodiment of the present application, an infrared thermal imager is installed on the vehicle. For example, the infrared thermal imager can be installed on a pan-tilt located at the position of the rear bumper of the vehicle. The infrared thermal imager is a device that generates a thermal image by detecting the infrared radiation emitted by an object, and is capable of converting the temperature distribution into a visible thermal image, where different colors or brightness represent different temperature regions. The infrared thermal imager can continuously collect multiple thermal images at a fixed frequency (such as 30 frames per second). In the thermal image, the pixel value of a pixel point represents the infrared radiation of the corresponding object in the environment.

[0046] The infrared thermal imager can communicate with the vehicle via the vehicle bus to transmit the collected thermal images to the vehicle's on-vehicle terminal or electronic control unit, so that the above modules can implement functions such as assisted driving, collision warning, or automatic obstacle avoidance based on environmental information such as thermal images. Hereinafter, the on-vehicle terminal or the electronic control unit will be simply referred to as the vehicle for illustration. After the vehicle obtains multiple thermal images collected by the infrared thermal imager, it can call an image recognition model to perform image recognition on each thermal image to obtain the thermal imaging information of each thermal image. Among them, the image recognition model can be a deep learning model trained with a large number of thermal images, which can effectively recognize and classify the targets in the thermal images. The image recognition model can be deployed on the on-vehicle terminal or on the server side, and the embodiments of the present application do not limit this. The thermal imaging information includes at least one of the category and size of the target object in the thermal image. Among them, the target object can be a static or dynamic obstacle, such as a pedestrian, an animal, etc.

[0047] 202. The vehicle determines the movement information of the target object based on the thermal imaging information of multiple thermal images. The movement information includes the movement direction and the movement speed.

[0048] In the embodiments of the present application, since the thermal imaging information can represent the target object in the thermal image, the movement direction and the movement distance of the target object can be determined based on the thermal imaging information of multiple thermal images arranged in chronological order. Furthermore, the movement speed of the target object can be determined based on the movement distance and the time difference, and the movement information of the target object can be obtained. Among them, the time difference can be the time interval between the acquisition times of two thermal images.

[0049] 203. The vehicle determines a first distance based on the ultrasonic radar. The first distance is the distance between the target object and the vehicle.

[0050] In the embodiments of the present application, after the vehicle detects the target object through the infrared thermal imager, it can detect the distance between the target object and the vehicle through the ultrasonic radar, that is, the first distance. Among them, the ultrasonic radar is a sensor that uses ultrasonic waves to detect the distance of an object. The ultrasonic radar calculates the distance between the object and the ultrasonic radar by emitting ultrasonic pulses and receiving their reflected echoes. Optionally, both the ultrasonic radar and the infrared thermal imager can be installed at the position of the rear bumper of the vehicle.

[0051] 204. The vehicle determines the danger level of the target object based on the thermal imaging information, movement information, and the first distance of the target object, and executes an obstacle avoidance strategy corresponding to the danger level.

[0052] In the embodiments of the present application, the vehicle can accurately evaluate the risk level of a target object based on the motion state of the target object and environmental information. Among them, the risk level is usually divided into multiple levels (such as low, medium, high). The higher the risk level, the greater the threat degree of the target object to the driving safety of the vehicle. In addition, the vehicle can execute corresponding obstacle avoidance strategies according to the risk level of the target object, so as to avoid the vehicle from colliding with the target object. The obstacle avoidance strategy is also the control operation taken by the vehicle to avoid the collision risk. For example, warning prompts, vehicle deceleration, steering avoidance, emergency braking, etc.

[0053] The embodiments of the present application provide a vehicle obstacle avoidance method, which can accurately and timely detect the obstacles existing around the vehicle and the risk level of the obstacles according to the thermal images collected by the infrared thermal imager and the distance information detected by the ultrasonic radar during the driving process, and then execute corresponding obstacle avoidance decisions to avoid the vehicle from colliding with the obstacles, improving the driving safety. In addition, the above method adopts a multi-sensor fusion strategy, which overcomes the limitations of a single sensor and further improves the accuracy of obstacle detection and vehicle obstacle avoidance.

[0054] Figure 3 is a flowchart of another vehicle obstacle avoidance method provided according to the embodiments of the present application. As Figure 3 shown, taking the execution of this method by the vehicle as an example, this method includes the following steps:

[0055] 301. The vehicle performs image recognition on multiple thermal images collected by the infrared thermal imager to obtain the thermal imaging information of each thermal image. The thermal imaging information includes at least one of the category and size of the target object in the thermal image.

[0056] In the embodiments of the present application, step 301 is the same as step 201 above and will not be elaborated here.

[0057] Optionally, before performing image recognition on the thermal image, the thermal image can also be preprocessed. Among them, the preprocessing includes but is not limited to: noise removal, contrast enhancement, size normalization, etc. By preprocessing the thermal image, the image quality of the thermal image can be improved, and further the accuracy of image recognition can be improved.

[0058] In addition, it should be supplemented and explained that the above step 201 is described by taking the invocation of an image recognition model to perform image recognition on a thermal image as an example. Optionally, the vehicle can also invoke multiple image recognition models to perform image recognition on multiple thermal images respectively, and obtain the thermal imaging information of the multiple thermal images output by each image recognition model. Among them, different image recognition models are used to recognize different types of obstacles, such as a pedestrian recognition model, an animal recognition model, etc. In addition, after obtaining the thermal imaging information output by the image recognition model, post-processing can also be performed on the thermal imaging information, such as non-maximum suppression (NMS, Non-Maximum Suppression), so as to remove redundant detection frames in the thermal imaging information and retain the detection frame that is most likely to represent the target object, which can reduce the repeated detection rate and improve the accuracy of target detection. By using different image recognition models to perform image recognition on multiple thermal images and post-processing the image recognition results, the accuracy of image recognition can be improved.

[0059] 302. The vehicle determines the movement information of the target object based on the thermal imaging information of multiple thermal images, where the movement information includes the movement direction and the movement speed.

[0060] In the embodiment of the present application, after the vehicle obtains multiple thermal images, it can determine the relative displacement of the target object in the thermal image according to the thermal imaging information of the multiple thermal images, and then determine the movement information of the target object, thereby providing data support for subsequent prediction of the movement trajectory of the target object, real-time tracking of the target object, assessment of the danger level of the target object, etc. The determination process of the movement information will be described separately below.

[0061] In some embodiments, the vehicle uses the inter-frame difference algorithm to determine the movement information of the target object. Among them, the inter-frame difference algorithm detects the moving area, that is, the area where the moving target object is located, by comparing the pixel differences of two consecutive frames or multiple frames of thermal images. Then, according to the thermal imaging information of the multiple thermal images, the target pixel points in the moving area in the thermal image, that is, the pixel points representing the target object, are determined; furthermore, according to the positions of the target pixel points in the multiple thermal images, the movement direction and the movement distance of the target pixel points are determined; then, according to the acquisition times of the multiple thermal images and the movement distance of the target pixel points, the movement speed of the target object is determined, so as to obtain the movement information of the target object. By using the inter-frame difference algorithm to determine the movement information of the target object, the movement situation of the target object can be determined according to the pixel change situation between multiple thermal images, and thus the movement information of the target object can be determined more accurately.

[0062] In some embodiments, the vehicle uses an optical flow estimation algorithm to determine the movement information of the target object. Among them, the optical flow estimation algorithm estimates the motion vector of each pixel by analyzing the changes in the pixel values of pixels in multiple thermal images. Its core idea is that the temperature distribution of the same object remains continuously changing between adjacent frames. Based on the thermal imaging information of multiple thermal images, the vehicle can determine the optical flow vector of the target pixel point. Then, according to the magnitude and direction of the optical flow vector, the movement information of the target object can be quickly determined. Among them, the magnitude of the optical flow vector can represent the moving speed of the target object, and the direction of the optical flow vector can represent the moving direction of the target object. By using the optical flow estimation algorithm to determine the movement information of the target object, the movement information of the target object can be determined more accurately according to the optical flow vector of the target pixel point, improving the accuracy of determining the movement information.

[0063] In addition, it should be noted that when the target object is a moving pedestrian, animal or other dynamic obstacle, as the target object moves, the target object may move out of the field of view of the infrared thermal imager, resulting in the infrared thermal imager being unable to capture the target object, and thus unable to effectively avoid obstacles for the target object. Therefore, after the vehicle determines the movement information of the target object, it can also predict the future movement trajectory of the target object according to the movement information, and then adaptively adjust the angle of the infrared thermal imager according to the movement trajectory to ensure that the infrared thermal imager can automatically track the target object. The angle adjustment process of the infrared thermal imager will be described below.

[0064] In some embodiments, after the vehicle determines the movement information of the target object, it predicts the movement trajectory of the target object in real time according to the movement information. Among them, the movement trajectory represents multiple positions of the target object within a future time period. For example, the movement trajectory represents the movement trajectory of the target object within the next 1 second. When the movement trajectory of the target object is outside the field of view of the infrared thermal imager, the vehicle can determine the angle adjustment parameters of the infrared thermal imager according to the movement information and movement trajectory of the target object. Optionally, the angle adjustment parameters include but are not limited to: pitch angle, yaw angle, angle adjustment rate, etc. Then, the vehicle adjusts the angle of the infrared thermal imager according to the angle adjustment parameters so that the field of view of the infrared thermal imager automatically tracks the target object. Optionally, since the infrared thermal imager is installed on the pan-tilt, the vehicle can accurately control the rotation of the pan-tilt according to the angle adjustment parameters, using the PID (Proportional-Integral-Derivative) control algorithm or other adaptive control algorithms, so as to adjust the pitch angle and yaw angle of the infrared thermal imager, ensuring that the lens of the infrared thermal imager can continuously aim at the target object. Even if the target object moves quickly, the infrared thermal imager can continuously track the target object.

[0065] For example, based on the relative position relationship between any predicted position beyond the field of view in the movement trajectory and the field of view, the vehicle determines the pitch angle and yaw angle of the infrared thermal imager. Then, based on the moving speed of the target object, the distance between the predicted position and the field of view, the vehicle determines the angle adjustment rate of the infrared thermal imager to ensure that the infrared thermal imager can smoothly track the target object, avoiding losing the target object due to too slow adjustment or causing image blurring due to too fast adjustment. In addition, considering the mechanical limitations of the pan-tilt head, such as the rotation range and the upper limit of the rotation rate of the pan-tilt head, the adjustment angle, angle adjustment rate, etc. determined by the vehicle cannot exceed the rotation range and the upper limit of the rotation rate of the pan-tilt head.

[0066] In some embodiments, when determining the angle adjustment parameters of the infrared thermal imager, the vehicle can also determine the relative speed between the target object and the vehicle based on the moving speed of the target object and the driving speed of the vehicle. Then, based on the moving information, movement trajectory, relative speed of the target object, and the distance between the target object and the vehicle, the vehicle determines the angle adjustment parameters of the infrared thermal imager. By further considering the relative speed and distance between the vehicle and the target object when determining the angle adjustment parameters of the infrared thermal imager, it can be ensured that the angle adjustment parameters of the infrared thermal imager can match the motion state of the target object in real time, and at the same time reserve sufficient reaction time for angle adjustment, avoiding the failure of the vehicle to avoid obstacles due to the delay in the adjustment of the infrared thermal imager.

[0067] Figure 4 is a flowchart of the angle adjustment of an infrared thermal imager provided according to an embodiment of the present application. As Figure 4 shown, when the vehicle detects a target object through the infrared thermal imager, it can determine the moving trend of the target object, predict the future position of the target object accordingly, and adjust the angle of the infrared thermal imager in real time to ensure that the infrared thermal imager can continuously track the moving target object.

[0068] 303. The vehicle determines a first distance based on the ultrasonic radar, where the first distance is the distance between the target object and the vehicle.

[0069] In the embodiment of the present application, the ultrasonic radar can be a high-precision ultrasonic sensor for measuring the distance between the vehicle and surrounding obstacles. The ultrasonic radar can be installed at positions such as the front bumper, rear bumper, and side of the vehicle, and the embodiment of the present application does not limit this. After the vehicle detects a target object through the infrared thermal imager, it can detect the distance between the target object and the vehicle through the ultrasonic radar, that is, obtain the first distance.

[0070] In addition to using ultrasonic radars for distance measurement, the vehicle can also perform distance measurement based on the thermal images collected by an infrared thermal imager, that is, infrared distance measurement, in order to obtain more accurate distance information as much as possible. The process of infrared distance measurement is described below through the following steps 304-305.

[0071] 304. The vehicle determines the parallax value of the target object at this moment based on two thermal images collected by two infrared thermal imagers at the same moment. The parallax value represents the relative displacement of the heat source center of the target object in the two thermal images.

[0072] In the embodiment of the present application, at least two infrared thermal imagers are installed on the vehicle. The vehicle acquires two thermal images collected by any two infrared thermal imagers at the same moment; then, the same heat source features, such as the heat source center or the heat source edge, are identified and matched in the two thermal images. Then, feature extraction algorithms such as SIFT (Scale-Invariant Feature Transform) are used to determine the parallax value of the target object at this moment. Among them, the parallax value reflects the relative displacement of the target object in the two thermal images.

[0073] 305. The vehicle determines a second distance based on the baseline distance, the parallax value, and the focal length of the infrared thermal imager. The second distance is the distance between the target object and the vehicle at this moment.

[0074] In the embodiment of the present application, the baseline distance represents the distance between the above two infrared thermal imagers. The baseline distance between any two infrared thermal imagers can be measured after the infrared thermal imagers are installed. Optionally, the vehicle can determine the distance between the target object and the vehicle according to the following formula (1).

[0075] Formula (1)

[0076] where B is the baseline distance, f is the focal length of the infrared thermal imager, d is the parallax value, and D is the second distance.

[0077] 306. The vehicle performs weighted summation on the first distance and the second distance to obtain the distance between the target object and the vehicle.

[0078] In the embodiment of the present application, the first distance is the distance obtained by measuring the target object with an ultrasonic radar, and the second distance is the distance obtained by infrared distance measurement. Therefore, when the infrared thermal imager captures a moving target object, the vehicle can use ultrasonic radar distance measurement and infrared distance measurement to respectively determine the distance between the target object and the vehicle, and fuse the distance measurement information of the ultrasonic radar and the infrared distance measurement information through a multi-sensor fusion algorithm to make up for the deficiencies of single-sensor distance measurement and improve the accuracy and stability of distance measurement.

[0079] Optionally, the weights of the first distance and the second distance may be set according to actual needs, for example, the sum of their weights is 1, which is not limited in the embodiment of the present application.

[0080] 307. The vehicle determines the danger level of the target object based on the thermal imaging information, movement information of the target object and the distance between the target object and the vehicle, and executes an obstacle avoidance strategy corresponding to the danger level.

[0081] In an embodiment of the present application, thermal imaging information can reflect the type and size of the target object, such as static obstacles, dynamic obstacles, large obstacles, small obstacles, etc. The movement information and the distance between the target object and the vehicle can reflect the speed at which the target object approaches or moves away from the vehicle. Therefore, combined with the above information, the vehicle can more accurately determine the danger level of the target object. Optionally, the danger level is positively correlated with the moving speed of the target object, and the danger level is negatively correlated with the distance between the target object and the vehicle. The danger level of dynamic obstacles is greater than that of static obstacles. The danger level includes low level, medium level and high level. After the vehicle determines the danger level of the target object, it can execute an obstacle avoidance strategy corresponding to the danger level according to the operating status of the vehicle. The following two cases are used as examples to illustrate this.

[0082] Case 1: The vehicle's driving mode is the automatic driving mode.

[0083] When the danger level of the target object is low, the vehicle can update the driving parameters, such as updating at least one of the driving speed and driving path, to avoid collision between the vehicle and the target object. For example, in an automatic parking scenario, when a low-level target object is detected, the vehicle can dynamically adjust the path planning and parking speed to avoid collision between the vehicle and the target object, thereby ensuring that the vehicle can park smoothly and accurately in the parking space.

[0084] When the danger level of the target object is medium, the vehicle outputs a prompt message through the vehicle screen and reduces the vehicle's speed. For example, the vehicle displays and broadcasts prompt messages through the vehicle screen to indicate that a medium-threat target object has been detected.

[0085] When the danger level of the target object is high, the vehicle outputs prompt information through the on-board large screen and triggers the vehicle's emergency braking to avoid impending collision accidents as much as possible, providing the last line of defense for driving safety.

[0086] Case 2: The vehicle's driving mode is manual driving mode.

[0087] When the danger level of the target object is low, the vehicle displays a prompt message through the on-board large screen, indicating that a low-threat target object is detected around the vehicle.

[0088] When the danger level of the target object is medium, the vehicle will display and broadcast prompt information through the on-board large screen to further remind the driver to pay attention to avoid the target object while driving.

[0089] When the danger level of the target object is high, the vehicle will display and broadcast prompt information through the on-board large screen and trigger the vehicle's emergency braking to avoid impending collision accidents as much as possible.

[0090] Therefore, in the case of manual driving by the driver, the vehicle can issue visual and auditory alarms in time when it detects target objects with collision risks around the vehicle, reminding the driver of potential dangers. If necessary, it can even autonomously trigger emergency braking to avoid impending collision accidents, providing the last line of safety for the driver and passengers.

[0091] Figure 5 FIG. 1 is a flowchart of an obstacle avoidance strategy according to an embodiment of the present application. Figure 5 As shown, after the vehicle detects relevant information of the target object based on the infrared thermal imager and ultrasonic radar, it can determine the danger level of the target object based on the relevant information and execute corresponding obstacle avoidance strategies to avoid collision between the vehicle and the target object.

[0092] The embodiment of the present application provides a vehicle obstacle avoidance method, which can accurately and timely detect obstacles around the vehicle and the danger level of obstacles based on the thermal image collected by the infrared thermal imager and the distance information detected by the ultrasonic radar during driving, and then execute corresponding obstacle avoidance decisions to avoid collision between the vehicle and the obstacle, thereby improving driving safety. In addition, the above method adopts a multi-sensor fusion strategy to overcome the limitations of a single sensor and further improve the accuracy of obstacle detection and vehicle obstacle avoidance.

[0093] Since the infrared thermal imager and ultrasonic radar are fused to detect target objects around the vehicle, the above method can maintain stable obstacle detection capabilities in low light, bad weather or complex driving conditions, and has high environmental adaptability and reliability.

[0094] In addition, compared with traditional vehicle obstacle avoidance solutions, the vehicle obstacle avoidance method provided by the embodiments of the present application can effectively detect high-speed moving or non-rigid obstacles, such as small animals running fast or floating obstacles, and can accurately predict the movement trajectories of dynamic obstacles, including their speeds, directions, and possible movement paths, so as to provide a forward-looking obstacle avoidance strategy for the vehicle, ensure that the vehicle can make fast and accurate obstacle avoidance decisions in a complex traffic environment, and can significantly reduce the potential collision risk in emergency obstacle avoidance scenarios, thereby effectively improving driving safety.

[0095] Figure 6 It is a block diagram of a vehicle obstacle avoidance device provided by an embodiment of the present application. This device is used to execute the steps when the above vehicle obstacle avoidance method is executed, such as Figure 6 As shown, the vehicle obstacle avoidance device includes: an identification module 601, a determination module 602, and an execution module 603.

[0096] The identification module 601 is used to obtain multiple thermal images collected by an infrared thermal imager, and respectively perform image recognition on the multiple thermal images to obtain the thermal imaging information of each thermal image. The thermal imaging information includes at least one of the category and size of the target object in the thermal image;

[0097] The determination module 602 is used to determine the movement information of the target object based on the thermal imaging information of the multiple thermal images. The movement information includes the movement direction and the transfer speed;

[0098] The determination module 602 is further used to determine a first distance based on an ultrasonic radar. The first distance is the distance between the target object and the vehicle;

[0099] The execution module 603 is used to determine the danger level of the target object based on the thermal imaging information, movement information, and the first distance of the target object, and execute an obstacle avoidance strategy corresponding to the danger level.

[0100] In some embodiments, the determination module 602 is used to determine the movement direction and movement distance of a target pixel point based on the thermal imaging information of the multiple thermal images. The target pixel point is the pixel point representing the target object in the thermal image; based on the movement direction and movement distance of the target pixel point, determine the movement information of the target object.

[0101] In some embodiments, the determination module 602 is used to determine the optical flow vector of a target pixel point based on the thermal imaging information of the multiple thermal images. The target pixel point is the pixel point representing the target object in the thermal image; based on the magnitude and direction of the optical flow vector, determine the movement information of the target object.

[0102] In some embodiments, the device further includes:

[0103] An adjustment module is configured to predict the movement trajectory of a target object based on the movement information of the target object, where the movement trajectory represents multiple positions of the target object within a future time period; in the case where the movement trajectory of the target object exceeds the field of view of the infrared thermal imager, determine the angle adjustment parameter of the infrared thermal imager based on the movement information and the movement trajectory of the target object; and adjust the angle of the infrared thermal imager based on the angle adjustment parameter so that the field of view of the infrared thermal imager automatically tracks the target object.

[0104] In some embodiments, the adjustment module is configured to determine the relative speed between the target object and the vehicle based on the movement information of the target object and the driving speed of the vehicle; and determine the angle adjustment parameter of the infrared thermal imager based on the movement information, the movement trajectory, the relative speed of the target object, and the distance between the target object and the vehicle.

[0105] In some embodiments, two infrared thermal imagers are installed on the vehicle, and the determination module 602 is further configured to determine the parallax value of the target object at a moment based on two thermal images collected by the two infrared thermal imagers at the same moment, where the parallax value represents the relative displacement of the heat source center of the target object in the two thermal images; determine the second distance based on the baseline distance, the parallax value, and the focal length of the infrared thermal imager, where the baseline distance represents the distance between the two infrared thermal imagers, and the second distance is the distance between the target object and the vehicle at the moment; and perform weighted summation on the first distance and the second distance to obtain the distance between the target object and the vehicle.

[0106] In some embodiments, the execution module 603 is configured to determine the danger level of the target object based on the thermal imaging information, the movement information of the target object, and the distance between the target object and the vehicle, where the danger level is positively correlated with the movement speed of the target object and negatively correlated with the distance between the target object and the vehicle.

[0107] In some embodiments, the danger level includes a low level, a medium level, and a high level; in the case where the driving mode of the vehicle is an autonomous driving mode, the execution module 603 is configured to update the driving parameters of the vehicle in the case where the danger level of the target object is at the low level, where the driving parameters include at least one of the driving speed and the driving path; output a prompt message through the in-vehicle large screen and reduce the driving speed of the vehicle in the case where the danger level of the target object is at the medium level; and output a prompt message through the in-vehicle large screen and trigger the emergency braking of the vehicle in the case where the danger level of the target object is at the high level.

[0108] In some embodiments, the danger levels include low level, medium level, and high level. When the driving mode of the vehicle is the manual driving mode, the execution module 603 is configured to, when the danger level of the target object is low, display a prompt message on the in-vehicle large screen, where the prompt message indicates that a target object is detected around the vehicle; when the danger level of the target object is medium, display and broadcast the prompt message on the in-vehicle large screen; and when the danger level of the target object is high, display and broadcast the prompt message on the in-vehicle large screen and trigger the emergency braking of the vehicle.

[0109] The embodiment of the present application provides a vehicle obstacle avoidance device, which can accurately and timely detect the obstacles existing around the vehicle and the danger levels of the obstacles according to the thermal images collected by the infrared thermal imager and the distance information detected by the ultrasonic radar during the driving process, and then execute corresponding obstacle avoidance decisions to avoid the vehicle from colliding with the obstacles, thereby improving the driving safety. In addition, the above method adopts a multi-sensor fusion strategy, which overcomes the limitations of a single sensor and further improves the accuracy of obstacle detection and vehicle obstacle avoidance.

[0110] It should be noted that when the vehicle obstacle avoidance device provided in the above embodiment runs the application program, only the division of the above function modules is used for illustration. In actual application, the above functions can be allocated to different function modules according to needs, that is, the internal structure of the terminal is divided into different function modules to complete all or part of the functions described above. In addition, the vehicle obstacle avoidance device provided in the above embodiment and the embodiment of the vehicle obstacle avoidance method belong to the same concept, and the implementation process can be seen in the method embodiment, which will not be elaborated here.

[0111] The embodiment of the present application also provides a computer-readable storage medium, in which at least one segment of computer program is stored, and the at least one segment of computer program is loaded and executed by a processor to implement the vehicle obstacle avoidance method in the above embodiment. For example, the computer-readable storage medium can be a Read-Only Memory (ROM), a Random Access Memory (RAM), a Compact Disc Read-Only Memory (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0112] The embodiment of the present application also provides a computer program product, including a computer program, and the computer program is executed by a processor to implement the vehicle obstacle avoidance method in the embodiment of the present application.

[0113] Those of ordinary skill in the art can understand that all or part of the steps to implement the above embodiments can be completed by hardware, or can be completed by a program instructing relevant hardware. The program can be stored in a computer-readable storage medium. The above-mentioned storage medium can be a read-only memory, a disk, an optical disc, etc.

[0114] The above are only alternative embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall be included within the protection scope of the present application.

Claims

1. A vehicle obstacle avoidance method, characterized in that: Applied to a vehicle, the vehicle is equipped with an infrared thermal imager and an ultrasonic radar, and the method comprises: Performing image recognition on the multiple thermal images collected by the infrared thermal imager to obtain thermal imaging information of each thermal image, wherein the thermal imaging information includes at least one of the category and size of the target object in the thermal image; Determining movement information of the target object based on the thermal imaging information of the plurality of thermal images, the movement information including a movement direction and a movement speed; Determine a first distance based on the ultrasonic radar, where the first distance is the distance between the target object and the vehicle; Based on the thermal imaging information, movement information and the first distance of the target object, a danger level of the target object is determined, and an obstacle avoidance strategy corresponding to the danger level is executed.

2. The method according to claim 1, characterized in that The step of determining the movement information of the target object based on the thermal imaging information of the plurality of thermal images comprises: Determine, based on the thermal imaging information of the plurality of thermal images, a moving direction and a moving distance of a target pixel point, the target pixel point being a pixel point representing the target object in the thermal image; Based on the moving direction and moving distance of the target pixel, the movement information of the target object is determined.

3. The method according to claim 1, characterized in that The step of determining the movement information of the target object based on the thermal imaging information of the plurality of thermal images comprises: Determine an optical flow vector of a target pixel point based on the thermal imaging information of the plurality of thermal images, wherein the target pixel point is a pixel point representing the target object in the thermal image; Based on the magnitude and direction of the optical flow vector, the movement information of the target object is determined.

4. The method according to claim 1, characterized in that: After determining the movement information of the target object, the method further includes: Predicting a movement trajectory of the target object based on the movement information of the target object, wherein the movement trajectory represents multiple positions of the target object in a future time period; When the moving track of the target object exceeds the field of view of the infrared thermal imager, determining an angle adjustment parameter of the infrared thermal imager based on the moving information of the target object and the moving track; Based on the angle adjustment parameter, the angle of the infrared thermal imager is adjusted so that the field of view of the infrared thermal imager automatically tracks the target object.

5. The method according to claim 4, characterized in that The step of determining the angle adjustment parameter of the infrared thermal imager based on the movement information and the movement trajectory of the target object includes: Determining a relative speed between the target object and the vehicle based on the movement information of the target object and the travel speed of the vehicle; An angle adjustment parameter of the infrared thermal imager is determined based on the movement information of the target object, the movement trajectory, the relative speed, and the distance between the target object and the vehicle.

6. The method according to claim 1, characterized in that The vehicle is equipped with two infrared thermal imagers, and the method further comprises: Based on two thermal images collected by the two infrared thermal imagers at the same time, determining a parallax value of the target object at the time, wherein the parallax value represents a relative displacement of a heat source center of the target object in the two thermal images; Determine a second distance based on a baseline distance, the parallax value, and a focal length of the infrared thermal imager, wherein the baseline distance represents the distance between the two infrared thermal imagers, and the second distance is the distance between the target object and the vehicle at the moment; A weighted sum is performed on the first distance and the second distance to obtain the distance between the target object and the vehicle.

7. The method according to claim 6, characterized in that The determining the danger level of the target object based on the thermal imaging information, the movement information and the first distance of the target object includes: Based on the thermal imaging information, movement information of the target object and the distance between the target object and the vehicle, the danger level of the target object is determined, wherein the danger level is positively correlated with the moving speed of the target object and negatively correlated with the distance between the target object and the vehicle.

8. The method according to claim 1, characterized in that The danger levels include low, medium and high; When the driving mode of the vehicle is the automatic driving mode, executing the obstacle avoidance strategy corresponding to the danger level includes: When the danger level of the target object is low, updating a driving parameter of the vehicle, the driving parameter including at least one of a driving speed and a driving path; When the danger level of the target object is medium, a prompt message is output through the vehicle-mounted large screen, and the driving speed of the vehicle is reduced; When the danger level of the target object is high, a prompt message is output through the vehicle-mounted large screen, and emergency braking of the vehicle is triggered.

9. The method according to claim 1, characterized in that: The danger levels include low, medium and high; When the driving mode of the vehicle is a manual driving mode, executing an obstacle avoidance strategy corresponding to the danger level includes: When the danger level of the target object is low, a prompt message is displayed on the vehicle-mounted large screen, and the prompt message indicates that the target object is detected around the vehicle; When the danger level of the target object is medium, the prompt information is displayed and broadcasted on the vehicle-mounted large screen; When the danger level of the target object is high, the prompt information is displayed and broadcasted through the vehicle-mounted large screen, and the emergency braking of the vehicle is triggered.

10. A vehicle obstacle avoidance device, characterized in that: Configured in a vehicle, the vehicle is equipped with an infrared thermal imager and an ultrasonic radar, and the device includes: A recognition module, used to perform image recognition on a plurality of thermal images collected by the infrared thermal imager to obtain thermal imaging information of each thermal image, wherein the thermal imaging information includes at least one of a category and a size of a target object in the thermal image; A determination module, configured to determine movement information of the target object based on the thermal imaging information of the plurality of thermal images, wherein the movement information includes a movement direction and a movement speed; The determination module is further configured to determine a first distance based on the ultrasonic radar, where the first distance is the distance between the target object and the vehicle; An execution module is used to determine the danger level of the target object based on the thermal imaging information, movement information and the first distance of the target object, and to execute an obstacle avoidance strategy corresponding to the danger level.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store at least one computer program, and the at least one computer program is used to execute the vehicle obstacle avoidance method described in any one of claims 1 to 9.

Images