Door opening anti-collision early warning dynamic fusion method and system based on recurrent neural network

Abstract

The invention relates to the technical field of automobile active safety, in particular to a door opening anti-collision early warning dynamic fusion method and system based on a recurrent neural network. The method comprises the following steps: S1, establishing a vehicle coordinate system; s2, first target data, second target data and image data detected by a millimeter wave radar, an ultrasonic radar and an all-round view camera are acquired respectively; s3, performing synchronization and normalization processing on the first target data, the second target data and the image data to obtain synchronous normalized data; s4, constructing an RNN depth recognition network; s5, constructing training data to pre-train the RNN depth recognition network; and S6, deploying the pre-trained RNN deep recognition network on the vehicle, and triggering door opening anti-collision early warning when the alarm confidence is lower than a preset threshold, so that the accuracy of alarm detection can be improved.

Description

Technical Field

[0001] This invention relates to the field of active safety technology for automobiles, specifically to a dynamic fusion method and system for door opening collision avoidance warning based on recurrent neural networks. Background Technology

[0002] With the rapid development of the global automotive industry and the continuous growth of residents' travel needs, the number of vehicles has experienced explosive growth, and the risk of traffic accidents has increased significantly. Among various traffic accidents, "door-opening kills" caused by opening car doors after parking are particularly typical and frequent. When drivers or passengers park in parking lots, roadside parking spaces, or temporary parking spots, they may be in a hurry to get out of the car or fail to observe carefully, causing the car door to suddenly open outwards, resulting in a violent collision with motor vehicles, non-motor vehicles, or pedestrians approaching rapidly from the side or rear of the vehicle, causing personal injury and economic losses.

[0003] The rapid development of sensor technology in recent years has provided strong technical support for the implementation of door opening collision warning systems. In particular, millimeter-wave radar, as the core technology of door opening collision warning systems, can detect targets within a range of tens of meters, and can work effectively in all weather conditions. It can monitor the distance, speed, and angle of targets behind the vehicle in real time, making it the preferred solution for implementing door opening collision warning systems.

[0004] In existing technologies, millimeter-wave radar is mainly used as the detection method for door opening collision avoidance functions. Millimeter-wave radar cannot distinguish between threatening and non-threatening targets, and may trigger false alarms due to stationary objects behind. For small targets such as children and pets, or targets made of non-metallic materials, the reflected signal of millimeter-wave radar is weak, and it is easy to miss detection in complex backgrounds. Radar waves may form false target signals after being reflected by the ground, walls, and other vehicles. There is a certain amount of time for radar waves to travel from transmission to reception. When a dangerous target at close range is obstructed, the target can quickly approach after the obstruction is removed, resulting in untimely warnings.

[0005] To overcome the aforementioned problems, some manufacturers have begun to explore multi-sensor collaborative detection solutions. These solutions fuse sensor data to reduce false alarms and false negatives, thereby improving the accuracy of the detection data. The most commonly used method is a pre-set fixed-weight scheme. This scheme leverages the detection characteristics of sensors in different scenarios to pre-set weights. For example, for the same target and the same state parameter (such as distance), the measurements from each sensor are multiplied by their corresponding weights and then summed. The advantages of the pre-set fixed-weight scheme are low computational complexity and simple engineering implementation; the disadvantages are poor environmental adaptability, decreased accuracy of sensors with high weights leading to deviations in the fusion results, and the continued use of weights when sensors malfunction, reducing the reliability of the fusion.

[0006] Neither mainstream millimeter-wave door collision avoidance solutions nor multi-sensor fusion solutions with preset fixed weights have fundamentally solved the risks of false alarms and missed alarms in collision avoidance systems, and their adaptability to complex scenarios is insufficient. Therefore, how to optimize multi-sensor fusion solutions is a key issue that needs to be addressed in the field of door collision avoidance technology. Summary of the Invention

[0007] The present invention aims to provide a dynamic fusion method and system for door opening anti-collision warning based on recurrent neural networks, which can improve the accuracy of alarm detection.

[0008] To address the aforementioned technical problems, the present invention provides the following technical solution: In a first aspect, the present invention provides a dynamic fusion method for door opening anti-collision warning based on a recurrent neural network, comprising the following steps: S1: Establish the vehicle coordinate system; S2: Acquire the first target data, second target data, and image data detected by the millimeter-wave radar, ultrasonic radar, and surround-view camera, respectively; S3: Synchronize and normalize the first target data, the second target data, and the image data to obtain synchronized and normalized data; S4: Construct an RNN deep recognition network; The output of the RNN deep recognition network includes alarm confidence, environmental confidence, and detected target information, while the input includes synchronized normalized data and the alarm confidence, environmental confidence, and target information from the previous recognition output; S5: Construct training data to pre-train the RNN deep recognition network; S6: Deploy the pre-trained RNN deep recognition network on the vehicle, and trigger the door opening collision avoidance warning when the alarm confidence level is lower than the preset threshold.

[0009] By adopting the above technical solution, and collecting, synchronizing, and normalizing vehicle-mounted image data, millimeter-wave data, and ultrasonic data, a recurrent neural network is used to train and output alarm levels. This eliminates the need for preset fixed thresholds. The introduction of an environmental confidence factor into the neural network achieves the following beneficial effects for better detection: First, sensor weights are no longer needed, avoiding the impact of sensor weight allocation on collision avoidance warning results; second, the introduction of an environmental confidence factor improves adaptability to complex scenarios; and finally, the step-by-step execution of target detection and alarm decision-making functions is eliminated, reducing algorithm computation, simplifying the door-opening collision avoidance decision logic, and improving system stability and reliability.

[0010] Optionally, the origin of the vehicle coordinate system in S1 is on the ground corresponding to the center of the license plate, the forward direction of the vehicle is the positive Y-axis direction, the right side of the vehicle is the positive X-axis direction, and the positive Z-axis direction is towards the air.

[0011] Optionally, the first target data in S2 includes a first timestamp, target position, target velocity, and target reflective area; the second target data includes a second timestamp and target detection distance; and the image data includes three signal data channels: rear, left, and right.

[0012] Optionally, S5 includes: S51: Acquire training data including multiple detection targets; S52: Calculate and sort the TTC indices of multiple detection targets, obtain the minimum TTC index, and transform the minimum TTC index into an alarm confidence level; S53: Use the image data from any of the surround-view cameras as the source of environmental confidence data in the training data, and obtain the environmental confidence based on deep networks or image processing methods.

[0013] Optionally, the alarm confidence calculation formula in S52 is: in, Represents the distance between the vehicle and the obstacle; This represents the absolute value of the relative speed between the vehicle and the obstacle. The obstacle here is not limited to static obstacles, but can also be a vehicle, pedestrian, or other moving object.

[0014] Optionally, the formula for calculating the environmental confidence level in S53 is: in, represent A single pixel in; Representative image The corresponding global threshold can be obtained using a preset value or an adaptive method. The value; This represents a function for calculating the percentage of image brightness.

[0015] Secondly, the present invention provides a door opening collision warning dynamic fusion system based on a recurrent neural network, used to realize the door opening collision warning dynamic fusion method based on a recurrent neural network as described in the first aspect, comprising: a controller, and a surround-view camera, a millimeter-wave radar, an ultrasonic radar, an audible and visual alarm device, and a door control mechanism respectively connected to the controller.

[0016] In summary, the present invention has at least the following beneficial technical effects: The method and system of this invention have no sensor weights and introduce an environmental confidence factor to improve environmental adaptability. Furthermore, target detection and alarm decision-making are integrated into one, and an end-to-end recurrent neural network is used, which can effectively improve the accuracy of alarm detection. Attached Figure Description

[0017] Figure 1 This is a flowchart of the dynamic fusion method for door opening anti-collision warning based on recurrent neural network in an embodiment of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0019] The terminology used in the following embodiments of the present invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in the specification and appended claims of the present invention, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in the present invention refers to and includes any or all possible combinations of one or more of the listed items. The terms “first” and “second” are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as “first” or “second” may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of the present invention, unless otherwise stated, “a plurality” means two or more.

[0020] This invention provides a dynamic fusion method for door opening anti-collision warning based on recurrent neural networks.

[0021] refer to Figure 1 A dynamic fusion method for door opening anti-collision warning based on recurrent neural networks includes the following steps: S1: Establish the vehicle coordinate system.

[0022] A vehicle coordinate system is established using a controller. In this embodiment, the origin of the vehicle coordinate system is on the ground corresponding to the exact center of the license plate. The vehicle's forward direction is the positive Y-axis, the right side of the vehicle is the positive X-axis, and the positive Z-axis is determined to be pointing towards the sky using the right-hand rule.

[0023] S2: Acquire the first target data, second target data, and image data detected by the millimeter-wave radar, ultrasonic radar, and surround-view camera respectively; the first target data includes the first timestamp, target position, target velocity, and target reflective area; the second target data includes the second timestamp and target detection distance; and the image data includes the rear, left, and right signal data.

[0024] The controller acquires the first target data detected by the millimeter-wave radar and retains the first timestamp, including The target location of each target Target speed Target reflective area Among them, the first The target location of each target is The target speed is The target reflective area is , .

[0025] The controller acquires the data of the second target detected by the ultrasonic radar and retains the second timestamp, thus obtaining the target detection distance of the ultrasonic radar as follows: .

[0026] The controller acquires image data detected by the surround-view cameras, including rear, left, and right signal data, respectively. , , .

[0027] S3: Synchronize and normalize the first target data, the second target data, and the image data to obtain synchronized and normalized data.

[0028] The sensor performs synchronization and normalization processing on the collected sensor data, which includes the aforementioned first target data. Second target data Image data , , .

[0029] S4: Construct an RNN deep recognition network; the output of the RNN deep recognition network includes alarm confidence, environmental confidence, and detected target information, and the input includes synchronous normalized data and the alarm confidence, environmental confidence, and target information output from the previous recognition.

[0030] The controller constructs an RNN deep recognition network with recognition capabilities. The output of the RNN deep recognition network includes alarm confidence. Environmental confidence The inputs include the detected target information, synchronous normalized data, and the alarm confidence level of the previous identification output. Environmental confidence And target information.

[0031] Target information includes: target location and corresponding target velocity. .

[0032] S5: Construct training data to pre-train the RNN deep recognition network.

[0033] The controller constructs training data to pre-train the RNN deep recognition network. The training data includes alarm confidence scores. Environmental confidence And target information.

[0034] Specifically, S5 includes the following steps: S51: Obtain publicly available fisheye camera datasets or perform fisheye adaptation on classic autonomous driving datasets to achieve coarse adjustment of target information; then obtain small batches of vehicle data to fine-tune the target information, resulting in... One detection target.

[0035] Coarse adjustment refers to using publicly available datasets, such as fisheye datasets created by companies like Valeo, or autonomous driving datasets like KITTI, but these are not in fisheye image format. Fisheye conversion algorithms are needed to process these image datasets into fisheye format.

[0036] Coarse tuning and fine tuning are just concepts; their essential actions are the same: building a network and then training it using a dataset. Publicly available datasets may not perfectly match the final training requirements in terms of image angle, color, white balance, and exposure. Their advantage is the large amount of data, which can be used to train a rough model. The small-batch dataset of vehicles, as a private dataset, is collected using the cameras ultimately used, making it more accurate than public datasets and used to improve the network's recognition accuracy.

[0037] S52: Calculation The Time to Collision (TTC) index of each detected target is calculated and ranked to obtain the minimum TTC index. This minimum TTC index is then transformed into an alarm confidence level. .

[0038] Alarm confidence The calculation formula is: in, Represents the distance between the vehicle and the obstacle; This represents the absolute value of the relative speed between the vehicle and the obstacle.

[0039] S53: Use the image data from any of the surround-view cameras as the source of environmental confidence data in the training data, and obtain the environmental confidence based on deep networks or image processing methods.

[0040] In this embodiment, the image is from the rear-view camera. As the source of environmental confidence, environmental confidence is obtained using color space transformation and brightness threshold. The calculation formula is as follows: in, represent A single pixel in; Representative image The corresponding global threshold can be obtained using a preset value or an adaptive method. The value; This represents a function for calculating the percentage of image brightness.

[0041] This invention does not consider the addition of an additional light sensor; therefore, a method is needed to detect the ambient brightness. This invention uses the brightness of an image to represent the ambient brightness. A brightness map can be obtained using color space transformation methods such as HSV, and finally... Statistical analysis of the percentage of brightness in an image.

[0042] S6: Deploy the pre-trained RNN deep recognition network on the vehicle, and trigger the door opening collision avoidance warning when the alarm confidence level is lower than the preset threshold.

[0043] During the use of RNN deep recognition networks, alarm confidence levels will be output. .

[0044] In this embodiment, alarm confidence level An alarm will be triggered when the value is ≤1.5. During actual use, the alarm confidence level can be adjusted. The output data is grouped to achieve multi-level alarms.

[0045] This invention also provides a dynamic fusion system for door opening anti-collision warning based on recurrent neural networks.

[0046] A dynamic fusion system for door opening collision warning based on recurrent neural networks includes: The surround-view camera is used to collect image data from both sides of the vehicle door and transmit it to the controller for processing in real time. In this embodiment, the vehicle-mounted left, right and rear surround-view cameras are mainly used to obtain their raw image data.

[0047] Millimeter-wave radars are installed on both sides of the rear bumper, respectively, to collect data on dangerous targets on the left and right rear sides.

[0048] An ultrasonic radar, mounted on the rear bumper, is used to collect data on nearby dangerous targets on the left and right sides.

[0049] Audible and visual alarm devices are used to alert and warn of dangerous targets.

[0050] The door control mechanism locks and unlocks the doors upon receiving a signal.

[0051] The controller collects and processes data on dangerous targets, implements early warning and control strategies, and outputs alarm signals and door control signals.

[0052] The controller is connected to the surround-view camera, millimeter-wave radar, ultrasonic radar, audible and visual alarm device, and door control mechanism to realize the above-mentioned dynamic fusion method for door opening collision avoidance warning based on recurrent neural network.

[0053] The various variations and specific examples of the methods provided in the above embodiments are also applicable to the door opening anti-collision warning dynamic fusion system based on recurrent neural networks in this embodiment. Through the foregoing detailed description of the door opening anti-collision warning dynamic fusion method based on recurrent neural networks, those skilled in the art can clearly understand the implementation method of the door opening anti-collision warning dynamic fusion system based on recurrent neural networks in this embodiment. For the sake of brevity, it will not be described in detail here.

[0054] The above description of the embodiments is only used to provide a detailed introduction to the technical solution of the present invention. However, the description of the above embodiments is only for the purpose of helping to understand the method and core idea of ​​the present invention, and should not be construed as a limitation of the present invention. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be covered within the protection scope of the present invention.

Claims

1. A dynamic fusion method for door opening anti-collision early warning based on recurrent neural networks, characterized in that, Includes the following steps: S1: Establish the vehicle coordinate system; S2: Acquire the first target data, second target data, and image data detected by the millimeter-wave radar, ultrasonic radar, and surround-view camera, respectively; S3: Synchronize and normalize the first target data, the second target data, and the image data to obtain synchronized and normalized data; S4: Construct an RNN deep recognition network; The output of the RNN deep recognition network includes alarm confidence, environmental confidence, and detected target information. The input includes synchronous normalized data and the alarm confidence, environmental confidence, and target information from the previous recognition output. S5: Construct training data to pre-train the RNN deep recognition network; S6: Deploy the pre-trained RNN deep recognition network on the vehicle, and trigger the door opening collision avoidance warning when the alarm confidence level is lower than the preset threshold.

2. The door opening anti-collision early warning dynamic fusion method based on recurrent neural network as described in claim 1, characterized in that, The origin of the vehicle coordinate system in S1 is on the ground corresponding to the center of the license plate. The direction of the vehicle's movement is the positive Y-axis, the right side of the vehicle is the positive X-axis, and the positive Z-axis points towards the sky.

3. The dynamic fusion method for door opening anti-collision early warning based on recurrent neural networks as described in claim 1, characterized in that, The first target data in S2 includes a first timestamp, target position, target velocity, and target reflective area; the second target data includes a second timestamp and target detection distance; and the image data includes three signal data channels: rear, left, and right.

4. The dynamic fusion method for door opening anti-collision early warning based on recurrent neural network as described in claim 3, characterized in that, S5 includes: S51: Acquire training data including multiple detection targets; S52: Calculate and sort the TTC indices of multiple detection targets, obtain the minimum TTC index, and transform the minimum TTC index into an alarm confidence level; S53: Use the image data from any of the surround-view cameras as the source of environmental confidence data in the training data, and obtain the environmental confidence based on deep networks or image processing methods.

5. The dynamic fusion method for door opening anti-collision early warning based on recurrent neural networks as described in claim 4, characterized in that, The formula for calculating the alarm confidence level in S52 is as follows: in, Represents the distance between the vehicle and the obstacle; This represents the absolute value of the relative speed between the vehicle and the obstacle.

6. The dynamic fusion method for door opening anti-collision early warning based on recurrent neural network as described in claim 4, characterized in that, The formula for calculating the environmental confidence level in S53 is as follows: in, represent A single pixel in; Representative image The corresponding global threshold can be obtained using a preset value or an adaptive method. The value; This represents a function for calculating the percentage of image brightness.

7. A dynamic fusion system for door opening anti-collision early warning based on recurrent neural networks, characterized in that, The method for implementing the door opening collision warning dynamic fusion method based on recurrent neural network as described in any one of claims 1-6 includes: a controller, and a surround view camera, a millimeter-wave radar, an ultrasonic radar, an audible and visual alarm device, and a door control mechanism respectively connected to the controller.

Images