Door collision warning system and method based on fisheye camera image distortion correction

Abstract

The present invention discloses a door collision warning system and method based on fisheye camera image distortion correction, which relates to the field of automobile safety reminder technology. The system uses a fisheye camera to collect scene images on both sides of the vehicle in a parking state in real time, uses an improved Harris corner detection algorithm to perform image calibration and distortion correction, combines a multi-task learning model of a convolutional neural network, and analyzes the potential collision risk in the door opening process in real time. The distance and height between the obstacle and the door are calculated by a signal processing module, and the reminder module generates reminder signals of different levels, which are fed back to the driver in time through the vehicle display and audio system, and the image display module marks the dangerous area and its dangerous level on the display device in real time. The present invention has a wider viewing angle coverage, higher detection accuracy and stronger environmental adaptability, effectively avoids the occurrence of door collision accidents, and is particularly suitable for small parking spaces, complex urban environments and high-density pedestrian areas.

Description

Technical Field

[0001] The present invention relates to the technical field of automobile safety reminders, and in particular to a door collision reminder system and method based on fisheye camera image distortion correction. Background Art

[0002] With the rapid development of the automobile industry and the acceleration of urbanization, the safety issues faced by vehicles during parking and when passengers get on and off the vehicle have become increasingly prominent. In narrow parking spaces or crowded areas, vehicle doors often collide with surrounding obstacles (such as walls, pillars, other vehicles or pedestrians) when opening. Such collisions may not only cause damage to the vehicle, but also cause serious injuries to passengers and pedestrians. Therefore, how to improve the safety of vehicle doors when opening and prevent collisions with surrounding obstacles has become a technical problem that needs to be solved urgently.

[0003] At present, most door collision warning systems on the market rely on radar sensors or ultrasonic sensors to monitor obstacles during the door opening process. Although these traditional sensors can provide certain obstacle detection functions, they have many shortcomings in practical applications. First, the detection performance of radar sensors is easily interfered by environmental factors, such as rain, fog and other bad weather conditions, resulting in insufficient stability and accuracy in complex environments. Secondly, the detection range of ultrasonic sensors is relatively limited, especially in the detection of small obstacles near the door, there is a significant blind spot, which makes it impossible for them to fully and accurately identify all potential dangers around the door.

[0004] Therefore, the existing door collision warning technology has significant deficiencies in detection accuracy, range and stability, and cannot provide sufficient safety protection in a changing environment. In order to solve these problems, a more efficient, accurate and stable door collision warning system is urgently needed, which can accurately identify and effectively warn of potential collision risks during the door opening process in complex environments. Summary of the invention

[0005] The purpose of the present invention is to provide a door collision warning system and method based on fisheye camera image distortion correction to solve the problems of blind spots, image distortion and insufficient multi-task recognition in the prior art, thereby improving the accuracy and reliability of door collision warning.

[0006] In order to achieve the above object, the present invention is implemented by the following technical solutions:

[0007] The door collision warning system based on fisheye camera image distortion correction includes:

[0008] An image acquisition module, including a fisheye camera installed under the rearview mirror of the vehicle, for real-time acquisition of scene images on both sides of the vehicle when the vehicle is parked;

[0009] The calibration module is used to calibrate the fisheye camera and determine the relationship between pixel coordinates, vehicle distance and target height through the improved Harris corner detection algorithm;

[0010] The target detection and recognition module is used to process the scene images on both sides collected by the image acquisition module, detect the category, position, size and height range of obstacles, and combine the multi-task learning model of the convolutional neural network (CNN) to use the RGB information and depth perception information of the image to simultaneously analyze the movement trajectory, speed and acceleration of the obstacles, so as to realize the linkage evaluation of the door status and the dynamic information of the obstacles;

[0011] A signal processing module, used to calculate the distance between the obstacle and the vehicle and the height of the obstacle according to the linkage evaluation result of the target detection and recognition module, and to determine the degree of influence of the obstacle on the opening of the door;

[0012] A reminder module, used to generate reminder signals of different levels according to the judgment result of the signal processing module;

[0013] The image display module is used to display the real-time collected scene images on both sides on the vehicle display device, and highlight the dangerous areas and their corresponding danger levels in a real-time annotation manner.

[0014] As a preferred solution of the present invention, the calibration module includes:

[0015] Checkerboard calibration block, place the checkerboard on both sides of the vehicle and parallel to the vehicle;

[0016] An image acquisition unit, which acquires the image of the calibration block through a fisheye camera and converts the image into a grayscale image;

[0017] A feature point detection unit extracts the pixel coordinates of feature points of the calibration block based on the Harris corner point detection algorithm;

[0018] The pixel area division unit is used to filter out the corner point coordinates of the door opening area, input the coordinates of the vehicle body coordinate system corresponding to the corner point area and the vehicle body width information, and adjust the calibration position of the door opening area. The calibration result can reflect the actual space coordinates when the door is open;

[0019] The mapping relationship fitting unit uses the least square method to fit the distance between the feature point pixel coordinates and the vehicle, and combines the feature point pixel coordinates of different height corner points of the calibration block to fit the mapping relationship between the pixel coordinates and the target height.

[0020] As a preferred solution of the present invention, the fisheye camera of the image acquisition module reuses the surround view camera system function of the vehicle to achieve full coverage of the door area through a 180° wide angle.

[0021] As a preferred solution of the present invention, the feature point detection unit adopts an improved Harris corner point detection algorithm to perform feature point detection, including the following steps:

[0022] By introducing the scale parameter Construct Gaussian kernel weight functions at different scales, perform weighted calculation on the gradient information of the calibration block image, and the correlation matrix The formula is:

[0023] ;

[0024] in, and The images are and Directional gradient; is the gradient The square of Directional strength; is the gradient The square of Directional strength; and is a cross term, indicating and Correlation of directional gradients; For scale Gaussian kernel weight function under , which is used to ensure the stability of corner detection to scale changes;

[0025] Based on the correlation matrix Calculate the response function , the formula is:

[0026] ;

[0027] in, is the determinant of the correlation matrix, is the trace of the correlation matrix, is a regularization parameter used to avoid the denominator being zero. An empirical coefficient for adjusting the strength of corner points;

[0028] According to the dynamic adjustment coefficient Setting the Threshold Filter the corner point position, the formula is:

[0029] ;

[0030] in, is the average of all response values, It is an empirical adjustment coefficient used to adapt to different image characteristics;

[0031] Based on the response function and threshold , the screening response value is greater than The pixel points are taken as corner points and the pixel coordinates of the feature points used for calibration are extracted.

[0032] As a preferred solution of the present invention, the target detection and recognition module includes:

[0033] An image input unit, used to receive the scene images on both sides acquired by the image acquisition module;

[0034] The obstacle detection unit is used to build a multi-task learning model based on the convolutional neural network (CNN), detect obstacles in the scene images on both sides, identify the category, location and size of the obstacles, and analyze the movement trajectory, speed and acceleration of the obstacles.

[0035] As a preferred solution of the present invention, the process of constructing a multi-task learning model based on a convolutional neural network CNN is as follows:

[0036] Data collection and preprocessing: Collect wide-angle fisheye images of static parking spaces and dynamic pedestrians approaching around car doors; eliminate image distortion through fisheye correction algorithm and convert the image into a standard perspective; expand the data set to add images under rainy, snowy and low-light conditions, provide diversified environmental data, and output corrected standardized images for subsequent model training;

[0037] Model construction and multi-task fusion: Generate depth maps based on corrected images, and design input layers compatible with fisheye images and depth maps; build a multi-task model, combine RGB images and depth information to complete obstacle category, position, size, and distance detection; add a dynamic obstacle trajectory analysis module to predict the next position and possible risks of moving targets, and realize door switch status detection at the same time, and output door status, speed, and acceleration information;

[0038] Interference and collision analysis module: Analyze the spatial relationship between the door and the obstacle, calculate the collision angle and force, and evaluate the potential collision risk; combine the door motion parameters with the obstacle position to predict the collision point; detect the spatial relationship between the door and the body, roof, and other parts to determine whether there is physical interference, and output the interference status and specific location;

[0039] Deployment and optimization: Optimize the computational process of the multi-task learning model and deploy it on the embedded device NVIDIA to achieve real-time detection capabilities; adjust the software and hardware operating parameters, utilize the computing resources of the embedded device, reduce energy consumption, support multi-task parallel processing, and meet the real-time performance requirements of ≥30FPS;

[0040] Testing and Verification: Test the detection performance in static parking scene and dynamic pedestrian approach, including detection accuracy and processing speed; verify the system's adaptability and stability to multiple dynamic targets in complex dynamic scenes where vehicle movement and door movement are synchronized, and evaluate the overall system performance.

[0041] As a preferred solution of the present invention, the signal processing module includes:

[0042] A distance calculation unit, used to calculate the distance between the obstacle and the vehicle based on the mapping relationship between the pixel coordinates and the vehicle distance provided by the calibration module and the pixel coordinates of the bottom boundary of the obstacle;

[0043] The height calculation unit is used to calculate the height of the obstacle based on the mapping relationship between pixel coordinates and height provided by the calibration module and the pixel coordinates of the top boundary of the obstacle. The formula is:

[0044] ;

[0045] in, Indicates the obstacle height obtained by preliminary calculation. and are the upper and lower boundary pixel coordinates of the obstacle in the image, Indicates the obstacle distance, is the focal length of the fisheye camera;

[0046] Height of obstacles To correct the error, the formula is:

[0047] ;

[0048] in, represents the corrected obstacle height, is the error correction value, and is the correction factor, is the pixel width of the obstacle in the image, is the actual width of the obstacle;

[0049] Dynamically adjust the corrected obstacle height , combined with historical calculation data for adjustment, the formula is:

[0050] ;

[0051] in, Indicates the estimated obstacle height at the current moment. is the estimated value at the previous moment, is the adjustment coefficient, which is used to assign weights to the current revised value and the historical value;

[0052] The interference judgment unit is used to judge the degree of influence of the obstacle on the opening of the door based on the distance between the obstacle and the vehicle and the height of the obstacle, and divide it into low danger, medium danger and high danger levels.

[0053] As a preferred solution of the present invention, the reminder module includes:

[0054] A signal generating unit, used to generate different levels of warning signals according to the degree of influence of obstacles on the opening of the door;

[0055] A low-risk prompt unit, used to output a low-frequency sound prompt when the impact level is a low-risk level;

[0056] A medium-risk warning unit, used to output a low-frequency sound warning and mark a yellow area when the impact level is a medium-risk level;

[0057] A high-risk prompt unit, used to output a high-frequency sound prompt and mark a red area when the impact level is a high-risk level;

[0058] The central control color marking unit displays green, yellow and red colors based on the vehicle display, indicating non-dangerous, medium-dangerous and high-dangerous areas respectively.

[0059] As a preferred solution of the present invention, the image display module includes:

[0060] A real-time image display unit, used to display scene images on both sides of the vehicle when it is parked;

[0061] A graphics overlay unit, used to overlay distance information between the obstacle and the door or body of the vehicle and a warning signal indication in the image;

[0062] Color-coded units are used to mark hazardous areas, with green indicating no hazard, yellow indicating medium hazard, and red indicating high hazard.

[0063] The reminder method of the door collision reminder system based on fisheye camera image distortion correction includes the following steps:

[0064] S1. Obtaining scene images on both sides of the scene when the vehicle is parked;

[0065] S2. Calibrate the pixel coordinates of the fisheye camera, and use the improved Harris corner detection algorithm to establish a mapping relationship between the pixel coordinates and the vehicle distance and target height, perform distortion correction on the fisheye image, and generate a corrected standard image;

[0066] S3. Process the corrected image to detect the category, location, size and height range of the obstacle, and combine the multi-task learning model of the convolutional neural network (CNN) to analyze the movement trajectory, speed and acceleration of the obstacle to generate dynamic information of the obstacle;

[0067] S4. Calculate the horizontal distance and actual height between the obstacle and the vehicle based on the boundary coordinates of the obstacle and the mapping relationship between the pixel coordinates and the actual distance and height;

[0068] S5. Based on the type, location, distance, height and dynamic information of the obstacle, determine the degree of impact of the obstacle on the door opening, assess the collision risk, and classify the risk into different levels;

[0069] S6. Generate a warning signal corresponding to the danger level according to the signal processing result, and mark the danger area and its level;

[0070] S7. Display the processed scene image and dangerous area on the vehicle display device, and update the dynamic status and warning information of the obstacle in real time.

[0071] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention adopts a fisheye camera as an image acquisition module, provides a 180° wide-angle field of view, can fully cover the door area and the surrounding environment, effectively avoids the visual blind spot problem of traditional radar or ultrasonic sensors, especially in small parking spaces or complex environments, and significantly improves the accuracy of collision reminders. Through the improved Harris corner detection algorithm, combined with precise image calibration and distortion correction, the image distortion problem of the fisheye camera is solved, and higher accuracy than traditional sensors is provided, especially when detecting small or low obstacles, it can effectively avoid missed detection. The target detection and recognition module adopts a multi-task learning model of a convolutional neural network CNN, combined with image RGB information and depth perception information, which can not only process static obstacles, but also monitor the motion trajectory, speed and acceleration of dynamic obstacles in real time, and enhance the system's ability to identify potential collision risks during the door opening process. Through image processing and depth information fusion, the system avoids the problem of performance degradation of radar and ultrasonic sensors in severe weather environments, and ensures stronger environmental adaptability and stability. Accurate distance calculation and height analysis enable the present invention to realize real-time and accurate reminder functions, generate reminder signals of different levels according to the relationship between obstacles and vehicle doors, and provide timely feedback to the driver to avoid collision risks and ensure the safety of vehicles and personnel. The wide-angle field of view of the fisheye camera effectively eliminates the blind spot problem in traditional technology and further improves the accuracy and real-time performance of the reminder signal. The present invention has obvious advantages in detection accuracy, real-time performance, environmental adaptability and system stability, and can significantly improve the safety of vehicles during parking and boarding and exiting. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. Among them:

[0073] Figure 1 It is a schematic diagram of the modular structure of the system of the present invention;

[0074] Figure 2 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0075] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution of the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings of the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all of the embodiments. Based on the described embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present invention.

[0076] The present invention provides a door collision warning system and method based on fisheye camera image distortion correction, aiming to solve the problems of detection blind spots, insufficient precision, environmental interference, etc. in traditional door collision warning systems, and provide an efficient, accurate and stable warning solution. This embodiment uses a fisheye camera to replace traditional radar and ultrasonic sensors. By combining image processing with depth perception technology, it can fully cover the door area and accurately identify and analyze the dynamic information of obstacles, thereby realizing effective warning of potential collision risks during the door opening process. The specific implementation methods of the present invention are described in detail below.

[0077] like Figure 1 As shown, an embodiment of the present invention provides a door collision warning system based on fisheye camera image distortion correction, including: an image acquisition module, a calibration module, a target detection and recognition module, a signal processing module, a warning module and an image display module.

[0078] (1) Image acquisition module

[0079] Used to collect scene images on both sides of the vehicle in real time when it is parked;

[0080] The image acquisition module includes a fisheye camera installed under the rearview mirror of the vehicle, which is mainly used to collect real-time scene images on both sides of the vehicle when it is parked. The fisheye camera has a 180° wide-angle field of view, which can monitor the environment around the door in all directions and transmit the image data to the subsequent image processing module in real time. The distortion correction of the fisheye camera is achieved through the subsequent processing module to ensure that the collected image can accurately reflect the position and distance of obstacles in the actual scene.

[0081] (2) Calibration module

[0082] Used to calibrate the fisheye camera and determine the relationship between pixel coordinates, vehicle distance and target height through the improved Harris corner detection algorithm;

[0083] The calibration module includes:

[0084] Checkerboard calibration block, place the checkerboard on both sides of the vehicle and parallel to the vehicle;

[0085] An image acquisition unit, which acquires the image of the calibration block through a fisheye camera and converts the image into a grayscale image;

[0086] A feature point detection unit extracts the pixel coordinates of feature points of the calibration block based on the Harris corner point detection algorithm;

[0087] The pixel area division unit is used to filter out the corner point coordinates of the door opening area, input the coordinates of the vehicle body coordinate system corresponding to the corner point area and the vehicle body width information, and adjust the calibration position of the door opening area. The calibration result can reflect the actual space coordinates when the door is open;

[0088] The mapping relationship fitting unit uses the least square method to fit the distance between the feature point pixel coordinates and the vehicle, and combines the feature point pixel coordinates of different height corner points of the calibration block to fit the mapping relationship between the pixel coordinates and the target height.

[0089] The calibration model can not only compensate for the distortion caused by the fisheye lens, but also provide basic data support for subsequent obstacle distance and height calculations.

[0090] In one embodiment, the feature point detection unit uses an improved Harris corner point detection algorithm to perform feature point detection, including the following steps:

[0091] By introducing the scale parameter Construct Gaussian kernel weight functions at different scales, perform weighted calculation on the gradient information of the calibration block image, and the correlation matrix The formula is:

[0092] ;

[0093] in, and The images are and Directional gradient; is the gradient The square of Directional strength; is the gradient The square of Directional strength; and is a cross term, indicating and Correlation of directional gradients; For scale Gaussian kernel weight function under , which is used to ensure the stability of corner detection to scale changes;

[0094] Based on the correlation matrix Calculate the response function , the formula is:

[0095] ;

[0096] in, is the determinant of the correlation matrix, is the trace of the correlation matrix, is a regularization parameter used to avoid the denominator being zero. An empirical coefficient for adjusting the strength of corner points;

[0097] According to the dynamic adjustment coefficient Setting the Threshold Filter the corner point position, the formula is:

[0098] ;

[0099] in, is the average of all response values, It is an empirical adjustment coefficient used to adapt to different image characteristics;

[0100] Based on the response function and threshold , the screening response value is greater than The pixel points are taken as corner points and the pixel coordinates of the feature points used for calibration are extracted.

[0101] (3) Target detection and recognition module

[0102] It is used to process the scene images on both sides collected by the image acquisition module, detect the category, position, size and height range of obstacles, and combine the multi-task learning model of the convolutional neural network (CNN) to use the RGB information and depth perception information of the image to simultaneously analyze the movement trajectory, speed and acceleration of the obstacles, so as to realize the linkage evaluation of the door status and the dynamic information of the obstacles;

[0103] An image input unit, used to receive the scene images on both sides acquired by the image acquisition module;

[0104] The obstacle detection unit is used to build a multi-task learning model based on the convolutional neural network (CNN), detect obstacles in the scene images on both sides, identify the category, location and size of the obstacles, and analyze the movement trajectory, speed and acceleration of the obstacles.

[0105] In another embodiment, the process of constructing a multi-task learning model based on a convolutional neural network CNN is as follows:

[0106] Data collection and preprocessing: Collect wide-angle fisheye images of static parking spaces and dynamic pedestrians approaching around car doors; eliminate image distortion through fisheye correction algorithm and convert the image into a standard perspective; expand the data set to add images under rainy, snowy and low-light conditions, provide diversified environmental data, and output corrected standardized images for subsequent model training;

[0107] Model construction and multi-task fusion: Generate depth maps based on corrected images, and design input layers compatible with fisheye images and depth maps; build a multi-task model, combine RGB images and depth information to complete obstacle category, position, size, and distance detection; add a dynamic obstacle trajectory analysis module to predict the next position and possible risks of moving targets, and realize door switch status detection at the same time, and output door status, speed, and acceleration information;

[0108] Interference and collision analysis module: Analyze the spatial relationship between the door and the obstacle, calculate the collision angle and force, and evaluate the potential collision risk; combine the door motion parameters with the obstacle position to predict the collision point; detect the spatial relationship between the door and the body, roof, and other parts to determine whether there is physical interference, and output the interference status and specific location;

[0109] Deployment and optimization: Optimize the computational process of the multi-task learning model and deploy it on the embedded device NVIDIA to achieve real-time detection capabilities; adjust the software and hardware operating parameters, utilize the computing resources of the embedded device, reduce energy consumption, support multi-task parallel processing, and meet the real-time performance requirements of ≥30FPS;

[0110] Testing and Verification: Test the detection performance in static parking scene and dynamic pedestrian approach, including detection accuracy and processing speed; verify the system's adaptability and stability to multiple dynamic targets in complex dynamic scenes where vehicle movement and door movement are synchronized, and evaluate the overall system performance.

[0111] (4) Signal processing module

[0112] Used to calculate the distance between the obstacle and the vehicle and the height of the obstacle according to the linkage evaluation result of the target detection and recognition module, and determine the degree of influence of the obstacle on the opening of the door;

[0113] A distance calculation unit, used to calculate the distance between the obstacle and the vehicle based on the mapping relationship between the pixel coordinates and the vehicle distance provided by the calibration module and the pixel coordinates of the bottom boundary of the obstacle;

[0114] The height calculation unit is used to calculate the height of the obstacle based on the mapping relationship between pixel coordinates and height provided by the calibration module and the pixel coordinates of the top boundary of the obstacle. The formula is:

[0115] ;

[0116] in, Indicates the obstacle height obtained by preliminary calculation. and are the upper and lower boundary pixel coordinates of the obstacle in the image, Indicates the obstacle distance, is the focal length of the fisheye camera;

[0117] Height of obstacles To correct the error, the formula is:

[0118] ;

[0119] in, represents the corrected obstacle height, is the error correction value, and is the correction factor, is the pixel width of the obstacle in the image, is the actual width of the obstacle;

[0120] Dynamically adjust the corrected obstacle height , combined with historical calculation data for adjustment, the formula is:

[0121] ;

[0122] in, Indicates the estimated obstacle height at the current moment. is the estimated value at the previous moment, is the adjustment coefficient, which is used to assign weights to the current revised value and the historical value;

[0123] The interference judgment unit is used to judge the degree of influence of the obstacle on the opening of the door based on the distance between the obstacle and the vehicle and the height of the obstacle, and divide it into low danger, medium danger and high danger levels.

[0124] The signal processing module further calculates the distance between the obstacle and the vehicle and the height of the obstacle based on the output information of the target detection and recognition module, and evaluates its impact on the door opening process. Specifically, the signal processing module calculates the horizontal distance between the obstacle and the door based on the mapping relationship provided by the calibration module, and determines the degree of its impact on the door opening based on the height information of the obstacle. Based on this calculation result, the system divides the collision risk into low, medium and high risk levels, providing a basis for the reminder module.

[0125] (5) Reminder module

[0126] Used to generate reminder signals of different levels according to the judgment results of the signal processing module;

[0127] A signal generating unit, used to generate different levels of warning signals according to the degree of influence of obstacles on the opening of the door;

[0128] A low-risk prompt unit, used to output a low-frequency sound prompt when the impact level is a low-risk level;

[0129] A medium-risk warning unit, used to output a low-frequency sound warning and mark a yellow area when the impact level is a medium-risk level;

[0130] A high-risk prompt unit, used to output a high-frequency sound prompt and mark a red area when the impact level is a high-risk level;

[0131] The central control color marking unit displays green, yellow and red colors based on the vehicle display, indicating non-dangerous, medium-dangerous and high-dangerous areas respectively.

[0132] The warning module generates warning signals of different levels based on the judgment results of the signal processing module. When the obstacle is close to the door and has a greater impact, the system will output a warning signal of a high-risk level to prompt the driver to take timely action; when the obstacle is at a longer distance and has a smaller impact, the system will output a warning signal of a low-risk or medium-risk level. The warning signal is prompted by low-frequency or high-frequency sound, and the dangerous area is marked with different colors on the vehicle display device so that the driver or passengers can identify and take measures in time.

[0133] (6) Image display module

[0134] It is used to display the real-time collected scene images on both sides on the vehicle display device, and highlight the dangerous areas and their corresponding danger levels in a real-time annotation manner.

[0135] A real-time image display unit, used to display scene images on both sides of the vehicle when it is parked;

[0136] A graphics overlay unit, used to overlay distance information between the obstacle and the door or body of the vehicle and a warning signal indication in the image;

[0137] Color-coded units are used to mark hazardous areas, with green indicating no hazard, yellow indicating medium hazard, and red indicating high hazard.

[0138] Through this module, the driver can see the obstacles around the door and their danger levels in real time, and can make corresponding operational decisions based on the displayed danger zone information. The system will also update the display content in real time according to the dynamic changes of obstacles to ensure the accuracy and timeliness of the reminder information.

[0139] like Figure 2 FIG. 1 is another embodiment of the present invention, which provides a reminder method of a door collision reminder system based on fisheye camera image distortion correction, comprising the following steps:

[0140] S1. Obtaining scene images on both sides of the scene when the vehicle is parked;

[0141] S2. Calibrate the pixel coordinates of the fisheye camera, and use the improved Harris corner detection algorithm to establish a mapping relationship between the pixel coordinates and the vehicle distance and target height, perform distortion correction on the fisheye image, and generate a corrected standard image;

[0142] S3. Process the corrected image to detect the category, location, size and height range of the obstacle, and combine the multi-task learning model of the convolutional neural network (CNN) to analyze the movement trajectory, speed and acceleration of the obstacle to generate dynamic information of the obstacle;

[0143] S4. Calculate the horizontal distance and actual height between the obstacle and the vehicle based on the boundary coordinates of the obstacle and the mapping relationship between the pixel coordinates and the actual distance and height;

[0144] S5. Based on the type, location, distance, height and dynamic information of the obstacle, determine the degree of impact of the obstacle on the door opening, assess the collision risk, and classify the risk into different levels;

[0145] S6. Generate a warning signal corresponding to the danger level according to the signal processing result, and mark the danger area and its level;

[0146] S7. Display the processed scene image and dangerous area on the vehicle display device, and update the dynamic status and warning information of the obstacle in real time.

[0147] The implementation process of the system of the present invention is as follows:

[0148] Image acquisition and preprocessing: The fisheye camera collects real-time scene images on both sides of the vehicle when it is parked, and the images are converted into standard perspective images through distortion correction.

[0149] Object detection and recognition: The fisheye camera is calibrated using the improved Harris corner detection algorithm, and obstacles are detected in the image through the CNN multi-task learning model to analyze the dynamic information of obstacles.

[0150] Distance and height calculation: Based on the mapping relationship provided by the calibration module, the signal processing module calculates the distance and height between the obstacle and the vehicle, and determines its impact on the door opening.

[0151] Collision risk assessment: The signal processing module assesses the collision risk and classifies the danger level based on the location, distance and height of the obstacle and the door opening status.

[0152] Warning signal generation and display: Based on the risk assessment results, the warning module generates warning signals of different levels and marks dangerous areas through sound prompts and display screens to provide drivers with real-time safety information.

[0153] Real-time monitoring and dynamic updates: The system continuously monitors the dynamic status of obstacles and updates reminder information in real time to ensure accuracy and timeliness in complex environments.

[0154] The door collision warning system of the present invention has the following significant advantages:

[0155] All-round monitoring: The 180° wide-angle field of view of the fisheye camera can fully cover the door area, avoiding the problem of traditional sensors being unable to detect at specific angles or blind spots.

[0156] High-precision detection: By combining image processing and depth perception, the ability to detect small obstacles is improved, solving the shortcomings of ultrasonic and radar sensors in detecting small obstacles.

[0157] Strong environmental adaptability: The system overcomes the sensitivity of traditional sensors to signal interference in harsh environments (such as rain, snow, fog and haze) through image distortion correction and depth information fusion, providing stable and reliable detection results.

[0158] Dynamic reminder: The system can predict the risk of collision and issue reminders in real time based on the dynamic characteristics of obstacles, ensuring the safety of vehicles and passengers can be effectively protected even in rapidly changing environments.

[0159] In summary, the present invention solves the problems of blind spots, insufficient detection accuracy and environmental interference in traditional door collision warning systems by using a fisheye camera for image acquisition, combined with precise distortion correction and target detection technology. By accurately identifying and dynamically analyzing obstacles through a multi-task learning model, the present invention not only improves the accuracy of door collision reminders, but also can operate stably in various complex environments, greatly enhancing the adaptability and real-time performance of the system. In addition, based on the wide-angle field of view and precise distance and height calculation of the fisheye camera, the present invention can generate different levels of reminder signals in real time, and promptly feedback the collision risk to the driver, thereby effectively preventing the occurrence of door collision accidents. In general, the present invention provides an efficient, accurate and stable door collision reminder solution with broad application prospects, especially in urban parking, narrow space driving and high-density pedestrian flow environments, which can significantly improve the safety of vehicles and protect the lives and property of drivers, passengers and pedestrians.

[0160] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.

[0161] Any process or method description in the flow chart or otherwise described herein can be understood to represent a module, fragment or portion of a code including one or more executable instructions for implementing the steps of a specific logical function or process. And the scope of the preferred embodiment of the present application includes other implementations, in which the functions may not be performed in the order shown or discussed, including in a substantially simultaneous manner or in a reverse order according to the functions involved.

[0162] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A door collision warning system based on fisheye camera image distortion correction, characterized in that: include: An image acquisition module, including a fisheye camera installed under the rearview mirror of the vehicle, for real-time acquisition of scene images on both sides of the vehicle when the vehicle is parked; The calibration module is used to calibrate the fisheye camera and determine the relationship between pixel coordinates, vehicle distance and target height through the improved Harris corner detection algorithm; The target detection and recognition module is used to process the scene images on both sides collected by the image acquisition module, detect the category, position, size and height range of obstacles, and combine the multi-task learning model of the convolutional neural network (CNN) to use the RGB information and depth perception information of the image to simultaneously analyze the movement trajectory, speed and acceleration of the obstacles, so as to realize the linkage evaluation of the door status and the dynamic information of the obstacles; A signal processing module, used to calculate the distance between the obstacle and the vehicle and the height of the obstacle according to the linkage evaluation result of the target detection and recognition module, and to determine the degree of influence of the obstacle on the opening of the door; A reminder module, used to generate reminder signals of different levels according to the judgment result of the signal processing module; An image display module is used to display the real-time collected scene images on both sides of the vehicle on the display device, and highlight the dangerous areas and their corresponding danger levels in a real-time annotation manner; The calibration module comprises: Checkerboard calibration block, place the checkerboard on both sides of the vehicle and parallel to the vehicle; An image acquisition unit, which acquires the image of the calibration block through a fisheye camera and converts the image into a grayscale image; A feature point detection unit extracts the pixel coordinates of feature points of the calibration block based on the Harris corner point detection algorithm; The pixel area division unit is used to filter out the corner point coordinates of the door opening area, input the coordinates of the vehicle body coordinate system corresponding to the corner point area and the vehicle body width information, and adjust the calibration position of the door opening area. The calibration result can reflect the actual space coordinates when the door is open; The mapping relationship fitting unit uses the least square method to fit the distance between the feature point pixel coordinates and the vehicle, combines the feature point pixel coordinates of different height corners of the calibration block, and fits the mapping relationship between the pixel coordinates and the target height; The feature point detection unit uses an improved Harris corner point detection algorithm to perform feature point detection, including the following steps: By introducing the scale parameter Construct Gaussian kernel weight functions at different scales, perform weighted calculation on the gradient information of the calibration block image, and the correlation matrix The formula is: ; in, and The images are and Directional gradient; is the gradient The square of Directional strength; is the gradient The square of Directional strength; and is a cross term, indicating and Correlation of directional gradients; For scale Gaussian kernel weight function under , which is used to ensure the stability of corner detection to scale changes; Based on the correlation matrix Calculate the response function , the formula is: ; in, is the determinant of the correlation matrix, is the trace of the correlation matrix, is a regularization parameter used to avoid the denominator being zero. An empirical coefficient for adjusting the strength of corner points; According to the dynamic adjustment coefficient Setting the Threshold Filter the corner point position, the formula is: ; in, is the average of all response values, It is an empirical adjustment coefficient used to adapt to different image characteristics; Based on the response function and threshold , the screening response value is greater than The pixel points are taken as corner points and the pixel coordinates of the feature points used for calibration are extracted.

2. The door collision warning system based on fisheye camera image distortion correction according to claim 1, characterized in that: The fisheye camera of the image acquisition module reuses the surround view camera system function of the vehicle to achieve full coverage of the door area through a 180° wide angle.

3. The door collision warning system based on fisheye camera image distortion correction according to claim 1, characterized in that: The target detection and recognition module includes: An image input unit, used to receive the scene images on both sides acquired by the image acquisition module; The obstacle detection unit is used to build a multi-task learning model based on the convolutional neural network (CNN), detect obstacles in the scene images on both sides, identify the category, location and size of the obstacles, and analyze the movement trajectory, speed and acceleration of the obstacles.

4. The door collision warning system based on fisheye camera image distortion correction according to claim 1, characterized in that: The process of constructing a multi-task learning model based on a convolutional neural network CNN is as follows: Data collection and preprocessing: Collect wide-angle fisheye images of static parking spaces and dynamic pedestrians approaching around car doors; eliminate image distortion through fisheye correction algorithm and convert the image into a standard perspective; expand the data set to add images under rainy, snowy and low-light conditions, provide diversified environmental data, and output corrected standardized images for subsequent model training; Model construction and multi-task fusion: Generate depth maps based on corrected images, and design input layers compatible with fisheye images and depth maps; build a multi-task model, combine RGB images and depth information to complete obstacle category, position, size, and distance detection; add a dynamic obstacle trajectory analysis module to predict the next position and possible risks of moving targets, and realize door switch status detection at the same time, and output door status, speed, and acceleration information; Interference and collision analysis module: Analyze the spatial relationship between the door and the obstacle, calculate the collision angle and force, and evaluate the potential collision risk; combine the door motion parameters with the obstacle position to predict the collision point; detect the spatial relationship between the door and the body, roof, and other parts to determine whether there is physical interference, and output the interference status and specific location; Deployment and optimization: Optimize the computational process of the multi-task learning model and deploy it on the embedded device NVIDIA to achieve real-time detection capabilities; adjust the software and hardware operating parameters, utilize the computing resources of the embedded device, reduce energy consumption, support multi-task parallel processing, and meet the real-time performance requirements of ≥30FPS; Testing and Validation: Testing detection performance in static scene parking spaces and dynamic scene pedestrian approaches, including detection accuracy and processing speed; In complex dynamic scenarios where the vehicle moves and the door moves synchronously, the system's adaptability and stability to multiple dynamic targets are verified, and the overall system performance is evaluated.

5. The door collision warning system based on fisheye camera image distortion correction according to claim 1, characterized in that: The signal processing module comprises: A distance calculation unit, used to calculate the distance between the obstacle and the vehicle based on the mapping relationship between the pixel coordinates and the vehicle distance provided by the calibration module and the pixel coordinates of the bottom boundary of the obstacle; The height calculation unit is used to calculate the height of the obstacle based on the mapping relationship between pixel coordinates and height provided by the calibration module and the pixel coordinates of the top boundary of the obstacle. The formula is: ; in, Indicates the obstacle height obtained by preliminary calculation. and are the upper and lower boundary pixel coordinates of the obstacle in the image, Indicates the obstacle distance, is the focal length of the fisheye camera; Height of obstacles To correct the error, the formula is: ; in, represents the corrected obstacle height, is the error correction value, and is the correction factor, is the pixel width of the obstacle in the image, is the actual width of the obstacle; Dynamically adjust the corrected obstacle height , combined with historical calculation data for adjustment, the formula is: ; in, Indicates the estimated obstacle height at the current moment. is the estimated value at the previous moment, is the adjustment coefficient, which is used to assign weights to the current revised value and the historical value; The interference judgment unit is used to judge the degree of influence of the obstacle on the opening of the door based on the distance between the obstacle and the vehicle and the height of the obstacle, and divide it into low danger, medium danger and high danger levels.

6. The door collision warning system based on fisheye camera image distortion correction according to claim 1, characterized in that: The reminder module comprises: A signal generating unit, used to generate different levels of warning signals according to the degree of influence of obstacles on the opening of the door; A low-risk prompt unit, used to output a low-frequency sound prompt when the impact level is a low-risk level; A medium-risk warning unit, used to output a low-frequency sound warning and mark a yellow area when the impact level is a medium-risk level; A high-risk prompt unit, used to output a high-frequency sound prompt and mark a red area when the impact level is a high-risk level; The central control color marking unit displays green, yellow and red colors based on the vehicle display, indicating non-dangerous, medium-dangerous and high-dangerous areas respectively.

7. The door collision warning system based on fisheye camera image distortion correction according to claim 1, characterized in that: The image display module comprises: A real-time image display unit, used to display scene images on both sides of the vehicle when it is parked; A graphics overlay unit, used to overlay distance information between the obstacle and the door or body of the vehicle and a warning signal indication in the image; Color-coded units are used to mark hazardous areas, with green indicating no hazard, yellow indicating medium hazard, and red indicating high hazard.

8. The reminder method of the door collision reminder system based on fisheye camera image distortion correction according to any one of claims 1 to 7, characterized in that: The following steps are involved: S1. Obtaining scene images on both sides of the scene when the vehicle is parked; S2. Calibrate the pixel coordinates of the fisheye camera, and use the improved Harris corner detection algorithm to establish a mapping relationship between the pixel coordinates and the vehicle distance and the target height, perform distortion correction on the fisheye image, and generate a corrected standard image; S3. Process the corrected image to detect the category, location, size and height range of the obstacle, and combine the multi-task learning model of the convolutional neural network (CNN) to analyze the movement trajectory, speed and acceleration of the obstacle to generate dynamic information of the obstacle; S4. Calculate the horizontal distance and actual height between the obstacle and the vehicle based on the boundary coordinates of the obstacle and the mapping relationship between the pixel coordinates and the actual distance and height; S5. Based on the type, location, distance, height and dynamic information of the obstacle, determine the degree of impact of the obstacle on the door opening, assess the collision risk, and classify the risk into different levels; S6. Generate a warning signal corresponding to the danger level according to the signal processing result, and mark the danger area and its level; S7. Display the processed scene image and dangerous area on the vehicle display device, and update the dynamic status and warning information of the obstacle in real time.

Images