Vehicle door opening early warning method and device, vehicle, storage medium and program product
By acquiring multiple frames of time-series images from the vehicle and using the YOLO model to identify targets, combined with sensors to determine the people inside the vehicle and the vehicle speed, and dynamically displaying warning information, the problem of recognition accuracy and adaptability when the vehicle door is opened is solved, reducing the risk of accidents.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHERY AUTOMOBILE CO LTD
- Filing Date
- 2026-01-07
- Publication Date
- 2026-05-01
AI Technical Summary
Existing technologies have low accuracy in target recognition when vehicle doors are opened and poor scene adaptability, which makes collision accidents more likely to occur when doors are opened.
By acquiring multiple frames of time-series images around the vehicle, a pre-trained YOLO model is used to identify target types and bounding box coordinates, dynamically determine target trends, and control the dashboard and display screen to display warnings when the target approaches. Combined with infrared and pressure sensors, the system determines the number of people inside the vehicle and the vehicle speed, and issues a warning.
It improves the accuracy and scene adaptability of target recognition, reduces the incidence of traffic accidents, and reduces the risk of collision when opening doors through dynamic early warning measures.
Smart Images
Figure CN121947162A_ABST
Abstract
Description
Vehicle door opening warning methods, devices, vehicles, storage media, and software products Technical Field
[0001] This application relates to the field of vehicle safety technology, and in particular to a vehicle door opening warning method, device, vehicle, storage medium, and program product. Background Technology
[0002] Vehicle door-opening accidents occur when occupants open car doors without properly checking for pedestrians or oncoming vehicles, causing collisions and potentially resulting in injuries or fatalities.
[0003] In related technologies, vehicle door opening safety largely relies on millimeter-wave radar to identify approaching targets, but its recognition accuracy is low, resulting in poor adaptability to different scenarios. Summary of the Invention
[0004] This application provides a vehicle door opening warning method, device, vehicle, storage medium, and program product to solve the problems of low recognition accuracy and poor scene adaptability in related technologies.
[0005] The first aspect of this application provides a vehicle door opening warning method, comprising the following steps: acquiring multiple frames of time-series images of a target area around the vehicle; inputting the multiple frames of time-series images into a pre-trained object recognition model, wherein the object recognition model outputs the types and bounding box coordinates of multiple targets in each frame of the multiple frames of time-series images; determining the running trend of at least one target based on the bounding box coordinates corresponding to the multiple targets in the multiple frames of time-series images; and when the running trend of at least one target is the target trend, controlling the vehicle's dashboard and display screen to display a door opening collision warning sign, and controlling the display screen to display target warning text, wherein the target warning text includes the type of the target.
[0006] Optionally, in one embodiment of this application, after the control display shows the target warning text, the method further includes: detecting the opening signal of the vehicle door; and after detecting the opening signal, controlling the vehicle to emit a target prompt sound.
[0007] Optionally, in one embodiment of this application, before acquiring a temporal image of the target area around the vehicle, the method further includes: detecting the pressure signal and infrared signal of the vehicle's seat; detecting the vehicle speed and the closing signal of the vehicle door; if the infrared signal, closing signal, and pressure signal are detected, and the vehicle speed is the target vehicle speed, then a temporal image of the target area around the vehicle is acquired; otherwise, a temporal image of the target area around the vehicle is not acquired.
[0008] Optionally, in one embodiment of this application, after the control display screen shows the target warning text, the method further includes: detecting the closing signal of each door of the vehicle and the vehicle speed; when the closing signal of each door is detected and the vehicle speed is greater than a preset vehicle speed, stopping the acquisition of time-series images of the target area around the vehicle.
[0009] Optionally, in one embodiment of this application, the object recognition model is a YOLO model. The YOLO model includes an input layer, a backbone network layer, a fusion layer, and an output layer. The input layer is used to process the temporal image; the backbone network layer is used to extract multi-scale image features from the processed temporal image; the fusion layer is used to fuse the multi-scale image features; and the output layer is used to output the type of at least one target and the coordinates of the target bounding box based on the fused multi-scale image features.
[0010] Optionally, in one embodiment of this application, determining the running trend of at least one target based on the target box coordinates of multiple targets in a multi-frame time sequence image includes: calculating the target box pixel size corresponding to the target based on the target box coordinates of the target in each frame time sequence image; if the target box pixel size increases with the time corresponding to the multi-frame time sequence image, then the movement trend of the target is determined to be the target trend, otherwise it is a non-target trend, wherein the target trend is moving closer to the vehicle.
[0011] A second aspect of this application provides a vehicle door opening warning device, comprising: an acquisition module for acquiring multiple frames of time-series images of a target area surrounding the vehicle; an input module for inputting the multiple frames of time-series images into a pre-trained object recognition model, wherein the object recognition model outputs the types and bounding box coordinates of multiple targets in each frame of the multiple frames of time-series images; a determination module for determining the running trend of at least one target based on the bounding box coordinates corresponding to the multiple targets in the multiple frames of time-series images; and a control module for controlling the vehicle's dashboard and display screen to display a door opening collision warning sign and controlling the display screen to display target warning text when the running trend of at least one target is a target trend, wherein the target warning text includes the type of the target.
[0012] Optionally, in one embodiment of this application, it further includes: a first detection module, used to detect the opening signal of the vehicle door after the target warning text is displayed on the control display screen; and to control the vehicle to emit a target prompt sound after the opening signal is detected.
[0013] Optionally, in one embodiment of this application, it further includes: a second detection module, used to detect the pressure signal and infrared signal of the vehicle's seat before acquiring a time-series image of the target area around the vehicle; detect the vehicle speed and the door closing signal; if the pressure signal, infrared signal, and closing signal are detected, and the vehicle speed is the target vehicle speed, then acquire a time-series image of the target area around the vehicle; otherwise, do not acquire a time-series image of the target area around the vehicle.
[0014] Optionally, in one embodiment of this application, it further includes: a third detection module, used to detect the closing signal of each door of the vehicle and the vehicle speed after the target warning text is displayed on the control display screen; when the closing signal of each door is detected and the vehicle speed is greater than a preset vehicle speed, the acquisition of time-series images of the target area around the vehicle is stopped.
[0015] Optionally, in one embodiment of this application, the object recognition model is a YOLO model. The YOLO model includes an input layer, a backbone network layer, a fusion layer, and an output layer. The input layer is used to process the temporal image; the backbone network layer is used to extract multi-scale image features from the processed temporal image; the fusion layer is used to fuse the multi-scale image features; and the output layer is used to output the type of at least one target and the coordinates of the target bounding box based on the fused multi-scale image features.
[0016] Optionally, in one embodiment of this application, the determining module is further configured to: calculate the pixel size of the target box corresponding to the target based on the target box coordinates in each frame of the time sequence image; if the pixel size of the target box corresponding to the target increases with the time corresponding to multiple frames of the time sequence image, then the motion trend of the target is determined to be the target trend, otherwise it is a non-target trend, wherein the target trend is to move closer to the vehicle.
[0017] A third aspect of this application provides a vehicle, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor executes the program to perform the vehicle door opening warning method as described in the above embodiments.
[0018] A fourth aspect of this application provides a computer-readable storage medium having a computer program or instructions stored thereon, which are executed by a processor to perform the vehicle door opening warning method as described above.
[0019] The fifth aspect of this application provides a computer program product, including a computer program or instructions, which, when executed, implement the vehicle door opening warning method as described in the above embodiments.
[0020] Therefore, this application has at least the following beneficial effects: The embodiments of this application can acquire multiple frames of time-series images of the target area around the vehicle, input these multiple frames of images into a pre-trained object recognition model, and the object recognition model outputs the types and bounding box coordinates of multiple targets in each frame of the time-series images. By recognizing targets through the object recognition model, the accuracy of target recognition is improved. Furthermore, the object recognition model can cover various vehicle usage scenarios, improving scenario adaptability. Based on the bounding box coordinates of multiple targets in the multiple frames of the time-series images, the running trend of at least one target is determined, achieving dynamic judgment of target movement trends. When the movement trend of at least one target matches the target trend, the vehicle's dashboard and display screen are controlled to display a door opening collision warning sign, and the display screen is controlled to display target warning text, thereby providing a warning to occupants of the vehicle when they open the door, reducing the incidence of traffic accidents. Thus, it solves the technical problems of low recognition accuracy and poor scenario adaptability in related technologies.
[0021] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description
[0022] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which: Figure 1 is a flowchart of a vehicle door opening warning method provided according to an embodiment of this application; Figure 2 is a structural diagram of a YOLOv5 model provided according to an embodiment of this application; Figure 3 is a schematic diagram of a vehicle door opening warning system provided according to an embodiment of this application; Figure 4 is a schematic diagram of the enabling principle of a vehicle door opening warning system provided according to an embodiment of this application; Figure 5 is a schematic diagram of the opening collision warning principle of a vehicle door opening warning system provided according to an embodiment of this application; Figure 6 is a schematic diagram of the closing principle of a vehicle door opening warning system provided according to an embodiment of this application; Figure 7 is a schematic diagram of the closing principle of a vehicle door opening warning system provided according to an embodiment of this application; Figure 8 is a schematic diagram of the structure of target recognition and proximity detection software provided according to an embodiment of this application; Figure 9 is an example diagram of a vehicle door opening warning device provided according to an embodiment of this application; Figure 10 is a structural schematic diagram of a vehicle provided according to an embodiment of this application. Detailed Implementation
[0023] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.
[0024] Before describing the solution of this application, let me first introduce some of the technical concepts involved in the solution of this application.
[0025] The development of artificial intelligence and automotive electronics technology has laid the foundation for the research and development of a car door opening warning system. In the field of artificial intelligence, there are many excellent object recognition model algorithms that can identify pedestrians, electric bicycles, motorcycles, and various cars on the road using high-definition cameras. Examples include R-CNN, YOLO, and SSD algorithms, with some models achieving an accuracy rate of over 95%. The vehicle body control domain is a crucial component of automotive domain controller technology, impacting the comfort of the vehicle's interior environment and the passenger's driving experience. The vehicle body domain control system integrates functions such as lighting control, wiper and washer control, central locking, window control, smart key, antenna drive, and network communication. With the development of new energy vehicle technology and the introduction of high-voltage power batteries, sufficient power support is provided for intelligent computing centers, whereas traditional cars cannot provide such support with their 12V batteries alone.
[0026] Therefore, this application provides a vehicle door opening warning method that combines artificial intelligence and vehicle body control technology to achieve vehicle door opening warning, thereby reducing the accident rate of door opening accidents.
[0027] Specifically, Figure 1 is a flowchart illustrating a vehicle door opening warning method provided in an embodiment of this application.
[0028] As shown in Figure 1, the vehicle door opening warning method includes the following steps: In step S101, multiple frames of time-series images of the target area around the vehicle are acquired.
[0029] The target area can be within 10m or 20m behind the vehicle, without specific limitations; multiple frames of time-series images can be obtained by capturing high-definition cameras with a resolution of no less than 1080P. The high-definition cameras can be set on the left and right rearview mirrors of the vehicle and take pictures periodically.
[0030] In some embodiments of this application, before acquiring a time-series image of the target area around the vehicle, the method further includes: detecting the pressure signal and infrared signal of the vehicle's seat; detecting the vehicle speed and the closing signal of the vehicle door; if the pressure signal, infrared signal, and closing signal are detected, and the vehicle speed is the target vehicle speed, then a time-series image of the target area around the vehicle is acquired; otherwise, a time-series image of the target area around the vehicle is not acquired.
[0031] Among them, the pressure signal can be detected by a pressure sensor, the infrared signal can be detected by an infrared sensor, and the target vehicle speed can be set to 0.
[0032] It is understood that the embodiments of this application can detect the pressure signal of the vehicle seat, the infrared signal, the vehicle speed, and the door closing signal. The pressure signal and the infrared signal are used to determine whether there is an occupant in the vehicle seat and whether there is a potential possibility of opening the door. The vehicle speed is used to determine whether the vehicle is stationary. The door closing signal is used to determine whether the door is closed. When the infrared signal, the closing signal, and the pressure signal are detected and the vehicle speed is the target speed, it indicates that there is someone in the vehicle and the vehicle is stationary, and there is a possibility that the person inside the vehicle will open the door. Therefore, a time-series image of the target area around the vehicle is acquired for subsequent door opening warning. Otherwise, the time-series image of the target area around the vehicle is not acquired.
[0033] It should be noted that, in this application, when vehicle speed is detected, it can first determine whether the vehicle speed is a valid signal. After determining that the vehicle speed is a valid signal, it can then determine whether the vehicle speed is the target vehicle speed, so as to avoid subsequent misjudgments caused by invalid vehicle speed.
[0034] Specifically, taking the driver's seat occupancy as an example, when pressure is detected in the driver's seat, a hardwired signal DriverSeatPressure=1 is sent to the BDM (Body Domain Module). Simultaneously, when the driver's side infrared sensor detects occupancy in the driver's seat, a hardwired signal DriverDoorInfDetection=1 is sent to the BDM. The BDM combines these two signals to determine that an occupant is in the driver's seat and there is a potential possibility of the driver's door being opened. At the same time, the BDM periodically receives the hardwired signal DriverDoorAjarSts=0, indicating that the driver's door is currently closed.
[0035] Next, it checks if the vehicle speed is 0. First, it checks if the vehicle speed valid signal ESP1_VehicleSpeedVSOSigValidData is 1, which indicates that the vehicle speed is valid. Then, it checks if the vehicle speed signal ABS_ESP_1_VehicleSpeedVSOSig is 0, which indicates that the vehicle is currently stationary. Combining the above five signals, the BDM determines that the preconditions for the door opening warning to be activated are met. Therefore, it sends an enable signal DOW_Enable_Sig = 1 to the ADCC (Advanced Driver Assistance System). The ADCC determines whether to activate the door opening warning based on the value of this signal.
[0036] In step S102, multiple frames of temporal images are input into a pre-trained object recognition model, and the object recognition model outputs the type and bounding box coordinates of multiple targets in each frame of the multi-frame temporal images.
[0037] The types of targets include pedestrians, electric vehicles (or motorcycles), and cars.
[0038] It is understood that, in the embodiments of this application, multiple frames of images can be input into a pre-trained object recognition model. The object recognition model outputs the type and bounding box coordinates of multiple targets in each frame of the multi-frame temporal image, thereby improving the accuracy of target recognition by recognizing targets through the object recognition model.
[0039] In some embodiments of this application, the object recognition model is a YOLO model. The YOLO model includes an input layer, a backbone network layer, a fusion layer, and an output layer. The input layer is used to process temporal images; the backbone network layer is used to extract multi-scale image features from the processed temporal images; the fusion layer is used to fuse multi-scale image features; and the output layer is used to output the type of at least one target and the coordinates of the target bounding box based on the fused multi-scale image features.
[0040] The YOLO model can specifically be the YOLOv5 model, which includes YOLOv5s, YOLOv5m, YOLOv5l, and YOLOv5x. These models have the same structure, but differ in the two parameters: depth_multiple (model depth) and width_multiple (model width). The object recognition model in this embodiment can specifically be the YOLOv5m model.
[0041] Specifically, the structure of the YOLOv5 model is shown in Figure 2, mainly consisting of four parts: input layer, backbone, neck, and head. The input layer includes Mosaic data augmentation, adaptive anchor box calculation, and adaptive image scaling; the backbone includes Focus structure and CSP structure; the neck is an FPN+PAN structure; and the head includes CIOU_Loss (Complete Crossover Ratio Loss).
[0042] In step S103, the running trend of at least one target is determined based on the target bounding box coordinates of multiple targets in the multi-frame time-series images.
[0043] In some embodiments of this application, determining the running trend of at least one target based on the target box coordinates of multiple targets in a multi-frame time-series image includes: calculating the target box pixel size corresponding to the target based on the target box coordinates of the target in each frame time-series image; if the target box pixel size increases with the time corresponding to the multi-frame time-series image, then the movement trend of the target is determined to be the target trend, otherwise it is a non-target trend, wherein the target trend is moving closer to the vehicle.
[0044] The target trend is to move closer to the vehicle, while the non-target trend can be to move away from the vehicle or remain stationary.
[0045] It is understood that, in this embodiment, the pixel size of the target bounding box corresponding to the target can be calculated based on the coordinates of the target bounding box in each frame of the temporal image. If the pixel size of the target bounding box increases with the time corresponding to multiple frames of the temporal image, the movement trend of the target is determined to be the target trend, i.e., gradually approaching the vehicle. Specifically, the object recognition model in this embodiment can embed an attention mechanism to identify pedestrians, electric bicycles, motorcycles, small cars, and large vehicles closest to the vehicle, and select and label the target type on the image, forming a new image and continuously sending it to the target judgment program. The target judgment program will judge the size and type of the selected box on the image. If it gradually increases, it is judged that a target such as a pedestrian, electric bicycle, motorcycle, small car, or large vehicle is approaching.
[0046] In step S104, when the movement trend of at least one target is a target trend, the instrument panel and display screen of the vehicle are controlled to display a door opening collision warning sign, and the display screen is controlled to display target warning text, wherein the target warning text includes the type of target.
[0047] The door opening collision warning sign can be a warning box, and the target warning text can be "XX (type of target) detected approaching, do not open the door".
[0048] It is understood that, in the embodiments of this application, when the movement trend of at least one target is the target trend, the instrument panel and display screen of the vehicle can be controlled to display a door opening collision warning sign, and the display screen can be controlled to display target warning text, so as to realize the warning of the occupants opening the door and reduce the incidence of traffic accidents.
[0049] In some embodiments of this application, after the control display shows the target warning text, the method further includes: detecting an opening signal of the vehicle door; and after detecting the opening signal, controlling the vehicle to emit a target warning sound.
[0050] The target alert sound is used to remind occupants of the vehicle who have opened the door that there is a risk of collision.
[0051] It is understood that, upon detecting a door opening signal, this embodiment controls the vehicle to emit a target prompt sound to remind occupants of the risk of collision when opening the door, thereby stopping the user from opening the door.
[0052] In some embodiments of this application, after the control display screen shows the target warning text, the method further includes: detecting the closing signal of each door of the vehicle and the vehicle speed; when the closing signal of each door is detected and the vehicle speed is greater than a preset vehicle speed, stopping the acquisition of time-series images of the target area around the vehicle.
[0053] The preset speed can be set according to specific circumstances, without any specific limitations, such as setting it to 15km / h or 20km / h.
[0054] It is understood that in this embodiment of the application, when the closing signal of each door of the vehicle and the vehicle speed are detected, if the closing signal of each door is detected and the vehicle speed is greater than the preset vehicle speed, it indicates that the vehicle is already in motion and there is no need for a door opening warning. Therefore, the acquisition of time-series images of the target area around the vehicle is stopped to reduce energy consumption.
[0055] Based on the above-mentioned vehicle door opening warning method, this application embodiment also provides a vehicle door opening warning system. The system can identify pedestrians, electric bicycles, motorcycles, tricycles and various types of cars approaching the vehicle within 10 meters behind the vehicle through high-definition cameras on the rearview mirrors on both sides of the vehicle body. If the above-mentioned targets approach, a warning box will pop up on the instrument panel and the central control screen before the occupants open the car door. If the car door is opened at this time, the audio system will immediately issue a warning.
[0056] The structure of the vehicle door opening warning system is shown in Figure 3. Specifically, it includes the ADCC intelligent driving control center, the BDM body domain controller and door lock system, output terminals (instrument panel, central control screen, audio system), left and right rearview mirror cameras, and input sensors (four-door infrared sensors, seat pressure sensors).
[0057] The basic principle of the vehicle door opening warning system is as follows: when the vehicle is stopped and there are still occupants inside, the cameras on both sides periodically take pictures of the rear of the vehicle and send the pictures to the ADCC intelligent driving control center via Ethernet communication link. The ADCC intelligent driving control center continuously receives the pictures with timestamps, identifies the targets in the pictures, and determines whether there are pedestrians, vehicles, etc. continuously approaching the vehicle. If the above phenomenon exists, ADCC will periodically send a door opening collision warning signal to the instrument panel and the central control screen. The instrument panel and the central control screen will continuously pop up a door opening collision warning sign and indicate that the approaching object is a pedestrian, electric bicycle, motorcycle, car, etc., reminding the occupants not to open the door. When occupants do not notice the prompts on the instrument panel and central control screen and are about to open the door, the BDM detects the door opening signal and sends a potential collision warning signal to the ADCC. After receiving the signal, the ADCC sends a door opening warning signal to the audio system. The audio system immediately emits a door opening collision warning tone, which includes an indication of what kind of object (i.e., target) is approaching from behind, reminding occupants not to continue opening the door. Once the camera detects that there is no gradually approaching object (i.e., target) within 10 meters behind, the door opening collision warning sign on the instrument panel and central control screen stops, and the door opening collision warning tone of the audio system is turned off.
[0058] The signals detected, transmitted, input, and output in the vehicle door opening warning system are shown in Table 1. Table 1 is a signal illustration table.
[0059]
[0060] 1. Left and right rearview mirror cameras.
[0061] The high-definition rearview mirror camera faces directly behind the vehicle, capable of capturing images within 20 meters behind, with a resolution of at least 1080p. When the vehicle is stationary and occupants are present, the camera initiates periodic image capture every 100ms, periodically transmitting the image data to the ADCC intelligent driving control center via LMirrorCameraPic and RMirrorCameraPic messages. The left and right rearview mirror cameras communicate with the ADCC via Ethernet.
[0062] 2. BDM and door lock systems.
[0063] The Body Design Manager (BDM) is responsible for controlling body-related functions and also serves as the central gateway for the entire vehicle. The BDM connects to and manages multiple CAN (Controller Area Network) segments, generally categorized as BD CAN (Body Domain Controller Area Network), IFT CAN (Integrated Function Test Controller Area Network), SUP CAN (Supervisor Controller Area Network), and ADAS CAN (Advanced Driver Assistance Systems Controller Area Network), etc. Each CAN segment carries different functional sensors or actuators. This system primarily involves four-door control; all four door controllers are mounted on the BDM's BD CAN segment and use CAN communication. The Advanced Driver Assistance Control Center (ADCC) is mounted on the ADAS CAN segment and uses CAN communication.
[0064] 3. ADCC Intelligent Driving Control Center.
[0065] ADCC (Advanced Driver Assistance Control Center) is the vehicle's intelligent driving controller, responsible for implementing Level 2 and above driver assistance functions. Compared to general controllers, ADCC has a more powerful and efficient central processing unit and larger data storage capacity, and typically uses operating systems such as Linux and QNX. To achieve the image object recognition function in this application, a fully trained YOLOv5 model is embedded in the controller, resulting in an object recognition accuracy of over 95%.
[0066] The ADCC intelligent driving control center periodically receives images from the left and right rearview mirror cameras, identifies the type of target in the images in real time, and determines whether the target is gradually approaching the camera by comparing the changes in the size of the target in consecutive images. If it is detected, it sends relevant signals to the BDM.
[0067] 4. Output terminals (instrument panel, central control screen, audio system).
[0068] When the instrument panel receives the ADCC2InstrumentLDoorCrashWarn and CloseObjectType signals from the ADCC, it should immediately display a clear pop-up: "Pedestrian / electric scooter / car / etc. is approaching on the left, prompting occupants not to open the left-side doors." When the instrument panel receives the ADCC2InstrumentRDoorCrashWarn signal from the ADCC, it should immediately display a clear pop-up: "Pedestrian / electric scooter / car / etc. is approaching on the right, prompting occupants not to open the right-side doors." The specific object type is determined based on the CloseObjectType signal.
[0069] When the central control screen receives the ADCC2IFTLDoorCrashWarn and CloseObjectType signals from the ADCC, it should immediately display a clear pop-up message: "Pedestrians / electric scooters / cars are approaching from the left; do not open the two left-side doors." Similarly, when the central control screen receives the ADCC2IFTRDoorCrashWarn and CloseObjectType signals from the ADCC, it should immediately display a clear pop-up message: "Pedestrians / electric scooters / cars are approaching from the right; do not open the two right-side doors." The specific object type is determined based on the CloseObjectType signal.
[0070] When the audio system receives the ADCC2AudioLDoorCrashWarn and CloseObjectType signals from the ADCC, it should immediately play an audio message: "A pedestrian / electric scooter / car / other object is approaching from the left; do not open the left-side doors." When the audio system receives the ADCC2AudioRDoorCrashWarn and CloseObjectType signals from the ADCC, it should immediately play an audio message: "A pedestrian / electric scooter / car / other object is approaching from the right; do not open the right-side doors." The specific object type is determined based on the CloseObjectType signal.
[0071] 5. Input sensors (four door infrared sensors and five seat pressure sensors).
[0072] The input sensors include infrared sensors on four doors and pressure sensors on five seats, used to determine whether there are occupants inside the vehicle. The infrared sensors on the doors detect the seats to determine if there is a living being on them. The seat pressure sensors determine if there is a load on the seat, but sometimes the seat contains an object instead of an occupant; in this case, combining the input from the infrared sensors can determine whether there is an occupant.
[0073] The program design of the vehicle door opening warning system of this application, that is, the implementation process of the vehicle door opening warning method of this application, is as follows.
[0074] 1. Door opening collision warning system activated.
[0075] As shown in Figure 4, the door opening collision warning system needs to be activated under certain conditions. Taking the driver's seat as an example, firstly, when pressure is detected in the driver's seat, a hardwired signal DriverSeatPressure=1 is sent to the BDM. Simultaneously, when the infrared sensor on the driver's side detects someone in the driver's seat, a hardwired signal DriverDoorInfDetection=1 is sent to the BDM. The BDM combines these two signals to determine that there is an occupant in the driver's seat and a potential possibility of opening the driver's door. At the same time, the BDM periodically receives the hardwired signal DriverDoorAjarSts=0, indicating that the driver's door is closed at this time.
[0076] To enable the door-opening collision warning system, the system first checks if the vehicle speed is 0. It then checks if the vehicle speed valid signal ESP1_VehicleSpeedVSOSigValidData is 1, indicating valid speed. Next, it checks if the vehicle speed signal ABS_ESP_1_VehicleSpeedVSOSig is 0, indicating the vehicle is stopped. Combining these five signals, the BDM determines that the system startup prerequisites are met and sends the system enable signal DOW_Enable_Sig = 1 to the ADCC. The ADCC then uses this signal value to determine whether to activate the system. This process is executed every 100ms.
[0077] 2. The door opening collision warning system activates the door opening collision warning.
[0078] As shown in Figure 5, taking the left side as an example, when the door opening collision warning system is enabled, i.e., the ADCC receives the signal DOW_Enable_Sig = 1, the ADCC first sends LMirrorCameraSts = 1 to the left rearview mirror camera to enable the camera to start working. The left rearview mirror camera will continuously send the captured images to the ADCC via Ethernet at a period of 100ms. The captured images are input into the ADCC's YoloV5 model for object recognition and object action judgment. When it detects that a vehicle or pedestrian is gradually approaching the vehicle within 10 meters, it sends LeftNearWarnSig = 1 to the ADCC. After receiving this signal, the ADCC immediately sends the ADCC2InstrumentLDoorCrashWarn = 1 and CloseObjectType signals to the instrument panel, and sends the ADCC2IFTLDoorCrashWarn = 1 and CloseObjectType signals to the central control screen. When the instrument panel and the central control screen receive the above signals, they immediately display a pop-up message: "A vehicle / pedestrian is approaching on the left side of the vehicle. Do not open the door!" If the occupant in the left seat does not notice the prompts on the instrument panel and large screen, the ADCC will receive the DriverDoorAjarSts=1 signal the moment the door is opened. The ADCC will then immediately send the ADCC2AudioLDoorCrashWarn=1 and CloseObjectType signals to the audio system. The audio system will then play a warning tone prohibiting the opening of the left door, instructing the occupant not to open the door.
[0079] 3. The door opening collision warning system is disabled.
[0080] As shown in Figure 6, after the door opening collision warning system activates the door opening collision warning, the left rear camera still sends images to the ADCC every 100ms. When the YOLOv5 object recognition model determines that there are no approaching vehicles or pedestrians behind, it will send the LeftNearWarnSig=0 signal to the ADCC. The ADCC then sends the following signals to the instrument panel and the central control screen respectively: ADCC2InstrumentLDoorCrashWarn=0 and ADCC2IFTLDoorCrashWarn=0. After receiving the signals, the instrument panel and the central control screen will no longer display the pop-up message "Vehicle / pedestrian is approaching on the left side of the vehicle, please do not open the door!". The ADCC also sends the signal ADCC2AudioLDoorCrashWarn=0 to the audio system at the same time, and the audio system immediately shuts off the warning sound.
[0081] 4. Turn off the door opening collision warning system.
[0082] As shown in Figure 7, the BDM continuously receives four gate closing signals, such as DriverDoorAjarSts=0.
[0083] When both the vehicle speed valid signal ESP1_VehicleSpeedVSOSigValidData = 1 and the vehicle speed judgment signal ABS_ESP_1_VehicleSpeedVSOSig > 15 are received, DOW_Enable_Sig = 0 will be sent to ADCC. When ADCC receives this signal, it will send LMirrorCameraSts = 0 and RMirrorCameraSts = 0 to the left and right rearview mirror cameras to turn off the cameras. If the YOLOv5 object recognition model does not receive an image from the camera within 5000ms, the model function will be shut down, thereby achieving the purpose of turning off the door collision warning system.
[0084] The ADCC intelligent driving control center in this application embodiment is equipped with a Linux system, and a trained YOLOv5m version model is deployed in the system. The model needs to achieve a vehicle and pedestrian recognition accuracy of over 95%.
[0085] As shown in Figure 8, which is a schematic diagram of the target recognition and proximity detection software, the left and right rearview mirror cameras send image data to the image preprocessing program. The image preprocessing program adds timestamps to the images, sorts them, and processes them into an image format that meets the input requirements of the YOLOv5 model. Then, the images are sent to the YOLOv5 model in order. The YOLOv5 model embeds an attention mechanism to identify pedestrians, electric bicycles, motorcycles, small cars, and large cars that are closest to the vehicle. It selects and labels the object type on the image, forming a new image and continuously sending it to the target judgment program. The target judgment program judges the size and type of the selection box on the image and sets the CloseObjectType signal value. If it gradually increases, it is judged that pedestrians, electric bicycles, motorcycles, small cars, large cars, etc. are approaching. It will trigger the LeftNearWarnSig / RightNearWarnSig=1 and CloseObjectType signal to be sent to the DA (Driver Assistance) Ethernet. The instrument panel, central control screen, and audio system will receive the signal and perform door opening warning processing.
[0086] According to the vehicle door opening warning method proposed in this application, multiple frames of time-series images of the target area around the vehicle can be acquired. These multiple frames of images are then input into a pre-trained object recognition model. The object recognition model outputs the types and bounding box coordinates of multiple targets in each frame of the multiple time-series images. By recognizing targets through the object recognition model, the accuracy of target recognition is improved. Furthermore, the object recognition model can cover various vehicle usage scenarios, improving scenario adaptability. Based on the bounding box coordinates of multiple targets in the multiple time-series images, the running trend of at least one target is determined, enabling dynamic judgment of target movement trends. When the movement trend of at least one target matches the target trend, the vehicle's dashboard and display screen are controlled to display a door opening collision warning sign, and the display screen is controlled to display target warning text, thereby providing a warning to occupants of the vehicle when they open the door and reducing the incidence of traffic accidents.
[0087] Next, the vehicle door opening warning device according to the embodiments of this application is described with reference to the accompanying drawings.
[0088] Figure 9 is a block diagram of a vehicle door opening warning device according to an embodiment of this application.
[0089] As shown in Figure 9, the vehicle door opening warning device 10 includes: an acquisition module 100, an input module 200, a determination module 300, and a control module 400.
[0090] The acquisition module 100 is used to acquire multiple frames of time-series images of the target area around the vehicle; the input module 200 is used to input the multiple frames of time-series images into a pre-trained object recognition model, and the object recognition model outputs the types and bounding box coordinates of multiple targets in each frame of the multiple time-series images; the determination module 300 is used to determine the running trend of at least one target based on the bounding box coordinates of multiple targets in the multiple time-series images; and the control module 400 is used to control the vehicle's dashboard and display screen to display a door opening collision warning sign and control the display screen to display target warning text when the running trend of at least one target is the target trend, wherein the target warning text includes the type of the target.
[0091] In some embodiments of this application, the vehicle door opening warning device 10 of this application embodiment further includes: a first detection module.
[0092] The first detection module is used to detect the opening signal of the vehicle door after the target warning text is displayed on the control display screen; after the opening signal is detected, the vehicle is controlled to emit a target prompt sound.
[0093] In some embodiments of this application, the vehicle door opening warning device 10 of this application embodiment further includes: a second detection module.
[0094] The second detection module is used to detect the pressure signal and infrared signal of the vehicle's seat before acquiring the time-series image of the target area around the vehicle; detect the vehicle speed and the door closing signal; if the pressure signal, infrared signal, and closing signal are detected, and the vehicle speed is the target speed, then the time-series image of the target area around the vehicle is acquired; otherwise, the time-series image of the target area around the vehicle is not acquired.
[0095] In some embodiments of this application, the vehicle door opening warning device 10 of this application embodiment further includes: a third detection module.
[0096] The third detection module is used to detect the closing signal of each door of the vehicle and the vehicle speed after the target warning text is displayed on the control display screen; when the closing signal of each door is detected and the vehicle speed is greater than the preset speed, the acquisition of time-series images of the target area around the vehicle is stopped.
[0097] In some embodiments of this application, the object recognition model is a YOLO model. The YOLO model includes an input layer, a backbone network layer, a fusion layer, and an output layer. The input layer is used to process temporal images; the backbone network layer is used to extract multi-scale image features from the processed temporal images; the fusion layer is used to fuse multi-scale image features; and the output layer is used to output the type of at least one target and the coordinates of the target bounding box based on the fused multi-scale image features.
[0098] In some embodiments of this application, the determining module 300 is further configured to: calculate the pixel size of the target box corresponding to the target based on the target box coordinates in each frame of the time sequence image; if the pixel size of the target box corresponding to the target increases with the time corresponding to multiple frames of the time sequence image, then the motion trend of the target is determined to be the target trend, otherwise it is a non-target trend, wherein the target trend is to move closer to the vehicle.
[0099] It should be noted that the foregoing explanation of the vehicle door opening warning method embodiment also applies to the vehicle door opening warning device of this embodiment, and will not be repeated here.
[0100] According to the vehicle door opening warning device proposed in this application embodiment, it can acquire multiple frames of time-series images of the target area around the vehicle, input the multiple frames of images into a pre-trained object recognition model, and the object recognition model outputs the type and target box coordinates of multiple targets in each frame of the multiple time-series images. By recognizing targets through the object recognition model, the accuracy of target recognition is improved, and the object recognition model can cover a variety of vehicle usage scenarios, improving scenario adaptability. Based on the target box coordinates corresponding to multiple targets in the multiple time-series images, the running trend of at least one target is determined, realizing dynamic judgment of target movement trend. When the movement trend of at least one target is the target trend, the device controls the vehicle's dashboard and display screen to display a door opening collision warning sign and controls the display screen to display target warning text, so as to realize the warning of the occupants opening the car door and reduce the incidence of traffic accidents.
[0101] Figure 10 is a schematic diagram of the structure of a vehicle provided in an embodiment of this application. The vehicle may include: a memory 1001, a processor 1002, and a computer program stored on the memory 1001 and executable on the processor 1002.
[0102] When the processor 1002 executes the program, it implements the vehicle door opening warning method provided in the above embodiments.
[0103] Furthermore, the vehicle also includes a communication interface 1003 for communication between the memory 1001 and the processor 1002.
[0104] The memory 1001 is used to store computer programs that can run on the processor 1002.
[0105] The memory 1001 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.
[0106] If the memory 1001, processor 1002, and communication interface 1003 are implemented independently, they can be interconnected via a bus to communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, only one thick line is used in Figure 10, but this does not indicate that there is only one bus or one type of bus.
[0107] Optionally, in a specific implementation, if the memory 1001, processor 1002, and communication interface 1003 are integrated on a single chip, then the memory 1001, processor 1002, and communication interface 1003 can communicate with each other through an internal interface.
[0108] The processor 1002 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.
[0109] This application also provides a computer-readable storage medium storing a computer program or instructions thereon, which, when executed by a processor, implements the vehicle door opening warning method described above.
[0110] This application also provides a computer program product, including a computer program or instructions, which, when executed, implement the vehicle door opening warning method described above.
[0111] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.
[0112] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.
[0113] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.
[0114] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or more of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (FPGAs), field-programmable gate arrays (FPGAs), etc.
[0115] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.
Claims
1. A vehicle door opening warning method, characterized in that, Includes the following steps: Acquire multi-frame temporal images of the target area surrounding the vehicle; The multi-frame temporal images are input into a pre-trained object recognition model, and the object recognition model outputs the type and bounding box coordinates of multiple targets in each frame of the multi-frame temporal images; The running trend of at least one target is determined based on the target bounding box coordinates corresponding to the multiple targets in the multi-frame time-series images; When the movement trend of at least one of the targets is a target trend, the instrument panel and display screen of the vehicle are controlled to display a door opening collision warning sign, and the display screen is controlled to display target warning text, wherein the target warning text includes the type of the target.
2. The vehicle door opening warning method according to claim 1, characterized in that, After controlling the display screen to show the target warning text, the method further includes: detecting the opening signal of the vehicle door; and after detecting the opening signal, controlling the vehicle to emit a target warning sound.
3. The vehicle door opening warning method according to claim 1, characterized in that, Before acquiring a time-series image of the target area around the vehicle, the method further includes: detecting the pressure signal and infrared signal of the vehicle's seat; detecting the vehicle's speed and the door closing signal; if the pressure signal, the infrared signal, and the closing signal are detected, and the vehicle speed is the target speed, then a time-series image of the target area around the vehicle is acquired; otherwise, a time-series image of the target area around the vehicle is not acquired.
4. The vehicle door opening warning method according to claim 1, characterized in that, After controlling the display screen to display the target warning text, the method further includes: detecting the closing signal of each door of the vehicle and the vehicle speed; when the closing signal of each door is detected and the vehicle speed is greater than a preset speed, stopping the acquisition of time-series images of the target area around the vehicle.
5. The vehicle door opening warning method according to claim 1, characterized in that, The object recognition model is a YOLO model, which includes an input layer, a backbone network layer, a fusion layer, and an output layer. The input layer is used to process the temporal image; the backbone network layer is used to extract multi-scale image features from the processed temporal image; the fusion layer is used to fuse the multi-scale image features; and the output layer is used to output the type of at least one target and the coordinates of the target bounding box based on the fused multi-scale image features.
6. The vehicle door opening warning method according to claim 1, characterized in that, The step of determining the running trend of at least one target based on the target box coordinates corresponding to the multiple targets in the multi-frame time-series images includes: calculating the target box pixel size corresponding to the target based on the target box coordinates corresponding to the target in each frame time-series image; if the target box pixel size corresponding to the target increases with the time corresponding to the multi-frame time-series images, then the movement trend of the target is determined to be a target trend, otherwise it is a non-target trend, wherein the target trend is to move closer to the vehicle.
7. A vehicle door opening warning device, characterized in that, include: The acquisition module is used to acquire multiple frames of time-series images of the target area around the vehicle; The input module is used to input the multi-frame temporal images into a pre-trained object recognition model, and the object recognition model outputs the type and bounding box coordinates of multiple targets in each frame of the multi-frame temporal images; The determination module is used to determine the running trend of at least one target based on the target bounding box coordinates corresponding to the multiple targets in the multi-frame time-series images; The control module is configured to control the vehicle's dashboard and display screen to display a door opening collision warning sign and control the display screen to display target warning text when the movement trend of at least one of the targets is a target trend, wherein the target warning text includes the type of the target.
8. A vehicle, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the program to implement the vehicle door opening warning method as described in any one of claims 1-6.
9. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, The computer program or instructions are executed by a processor to implement the vehicle door opening warning method as described in any one of claims 1-6.
10. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed, they implement the vehicle door opening warning method as described in any one of claims 1-6.