Vehicle door opening early warning method and controller

Abstract

The invention provides a vehicle door opening early warning method and a controller, relates to the technical field of automobiles, and can accurately predict the opening state of a vehicle door and perform early warning to reduce the risk of'door opening killing '. The method comprises the following steps: acquiring multi-modal data; the multi-modal data comprises first modal data, second modal data, third modal data and fourth modal data; wherein the first modal data is used for indicating the topological structure change condition of the vehicle; the second modal data is used for indicating the radial speed change condition when the vehicle door is opened; the third modal data is used for indicating a two-dimensional image associated with vehicle door opening; the fourth modal data is used for indicating the distance between the obstacle object and the vehicle door; inputting the multi-modal data into a vehicle door opening prediction model to obtain an output result; and determining different early warning levels according to the output result, and adopting different early warning strategies under different early warning levels.

Description

Technical Field

[0001] The present application relates to the field of automotive technology, and in particular to a vehicle door opening warning method and controller. Background Art

[0002] "Door-opening" accidents (i.e., collisions caused by the sudden opening of a car door) have become a major road safety risk due to their suddenness and severe consequences. Despite widespread concern about their harmful effects, existing door-opening warning technologies face significant challenges: insufficient door-opening recognition accuracy and overly simplistic processing methods. This combination of imprecision and simplification makes existing solutions incapable of reliably preventing "door-opening" accidents. Summary of the Invention

[0003] The purpose of this application is to provide a vehicle door opening warning method and controller, which can accurately predict the state of the door opening and issue a warning, thereby reducing the risk of "door opening kill".

[0004] To achieve the above objectives, this application adopts the following technical solutions:

[0005] In a first aspect, a vehicle door opening warning method is provided, the method comprising: acquiring multimodal data; the multimodal data comprising first modal data, second modal data, third modal data and fourth modal data; wherein the first modal data is used to indicate changes in the topological structure of the vehicle; the second modal data is used to indicate changes in radial velocity when the door is opened; the third modal data is used to indicate a two-dimensional image associated with the door opening; and the fourth modal data is used to indicate the distance between the obstacle object and the door; the multimodal data is input into a door opening prediction model to obtain an output result; different warning levels are determined according to the output result, and different warning strategies are adopted at different warning levels.

[0006] An embodiment of the present application provides a vehicle door opening warning method, comprising: acquiring multimodal data; the multimodal data comprising first, second, third, and fourth modal data; wherein the first modal data indicates changes in the vehicle's topological structure; the second modal data indicates changes in radial velocity when the door is opened; the third modal data indicates a two-dimensional image associated with the door opening; and the fourth modal data indicates the distance between the obstacle and the door; inputting the multimodal data into a door opening prediction model to obtain an output result; determining different warning levels based on the output result, and employing different warning strategies at different warning levels. It can be seen that the embodiment of the present application inputs the multimodal data into the door opening prediction model, determines different warning levels based on the output result, and then employs different warning strategies at different warning levels. On the one hand, the complementary nature of the multimodal data overcomes the bottleneck of single data, thereby improving the accuracy of door opening detection in complex scenarios. On the other hand, different warning stages are determined, and different warning strategies are then employed to further reduce the possibility of "door opening kill" events. Based on the above two aspects, accurate warning of door opening events can be provided, reducing the risk of "door opening kill" events.

[0007] Optionally, different warning strategies are adopted at different warning levels, including: at the first warning level, prompting the door to be opened through the display screen; at the second warning level, an alarm is sounded by the buzzer and the maximum angle of door opening is limited; at the third warning level, the door is locked and an emergency stop signal is broadcast; the emergency stop information is used to instruct the target vehicle to activate the automatic emergency braking system (AEB).

[0008] Optionally, the output results include: the door opening angle and the distance to the obstacle object; when the door opening angle is greater than the first angle threshold and less than the second angle threshold, and the distance to the obstacle object is greater than the first distance threshold, it is determined to be in the first warning level; when the door opening angle is greater than the first angle threshold, and the distance to the obstacle object is greater than the second distance threshold and less than the first distance threshold, it is determined to be in the second warning level; when the door opening angle is greater than the third angle threshold, and the distance to the obstacle object is less than the second distance threshold, it is determined to be in the third warning level; wherein, the third angle threshold is less than the first angle threshold.

[0009] Optionally, multimodal data is input into the door opening prediction model to obtain an output result, including: determining the weight of each sensor based on the environmental sensitivity coefficient and sensor health of the sensor corresponding to each modal data; based on the weight of each sensor, multimodal data is input into the door opening prediction model with corresponding weight to obtain an output result.

[0010] Optionally, determining the weight of each sensor according to the environmental sensitivity coefficient and sensor health of the sensor corresponding to each modal data includes: determining the weight of the sensor using the following expression;

[0011]

[0012] Among them, W i (t) represents the weight of the sensor, α i (t) represents the environmental sensitivity coefficient of the sensor, β i (t) represents the sensor health of the sensor.

[0013] Optionally, the vehicle door opening warning method further includes: detecting the current environment of the vehicle; determining the environmental sensitivity coefficient corresponding to the current environment from an environmental level table; wherein the environmental level table is used to indicate the effectiveness of sensors under various environments.

[0014] Optionally, the vehicle door opening warning method further includes: determining the sensor health of the sensor based on a historical false detection rate of the sensor.

[0015] Optionally, the door opening prediction model is a spatiotemporal network model; the spatiotemporal network model includes at least: an efficient reparameterized network (EfficientRep) backbone network, a spatiotemporal attention module, an optical flow trajectory tracking module and a long short-term memory network (LSTM) prediction module.

[0016] Optionally, the vehicle door opening warning method also includes: transmitting vehicle-to-everything (V2X) communication information to the target vehicle; the V2X communication information is used to indicate the door opening status; the V2X communication information includes at least one of the following: door identification, door opening angle, data confidence, and timestamp.

[0017] On the second aspect, a vehicle door opening warning device is also provided, including an acquisition module, a processing module and a warning module; the acquisition module is used to acquire multimodal data; the multimodal data includes first modal data, second modal data, third modal data and fourth modal data; wherein, the first modal data is used to indicate the change in the topological structure of the vehicle; the second modal data is used to indicate the change in radial velocity when the door is opened; the third modal data is used to indicate a two-dimensional image associated with the opening of the door; the fourth modal data is used to indicate the distance between the obstacle object and the door; the processing module is used to input the multimodal data into the door opening prediction model to obtain an output result; the warning module is used to determine different warning levels according to the output results, and adopt different warning strategies at different warning levels.

[0018] Optionally, the warning module is used to adopt different warning strategies at different warning levels, including: the warning module is specifically used to, at the first warning level, prompt the door to be opened through the display screen; the warning module is specifically used to, at the second warning level, alarm through the buzzer and limit the maximum angle of the door opening; the warning module is specifically used to, at the third warning level, lock the door and broadcast an emergency stop signal; the emergency stop information is used to instruct the target vehicle to activate the automatic emergency braking system (AEB).

[0019] Optionally, the output results include: the door opening angle and the obstacle object distance; the processing module is specifically used to determine that it is in the first warning level when the door opening angle is greater than the first angle threshold and less than the second angle threshold, and the obstacle object distance is greater than the first distance threshold; the processing module is specifically used to determine that it is in the second warning level when the door opening angle is greater than the first angle threshold and the obstacle object distance is greater than the second distance threshold and less than the first distance threshold; the processing module is specifically used to determine that it is in the third warning level when the door opening angle is greater than the third angle threshold and the obstacle object distance is less than the second distance threshold; wherein, the third angle threshold is less than the first angle threshold.

[0020] Optionally, the processing module is used to input the multimodal data into the door opening prediction model to obtain an output result, including: determining the weight of each sensor based on the environmental sensitivity coefficient and sensor health of the sensor corresponding to each modal data; and inputting the multimodal data into the door opening prediction model with the corresponding weight according to the weight of each sensor to obtain the output result.

[0021] Optionally, the processing module is configured to determine the weight of each sensor according to the environmental sensitivity coefficient and sensor health of the sensor corresponding to each modal data, including: determining the weight of the sensor using the following expression;

[0022]

[0023] Among them, W i (t) represents the weight of the sensor, α i (t) represents the environmental sensitivity coefficient of the sensor, β i (t) represents the sensor health of the sensor.

[0024] Optionally, the vehicle door opening warning method also includes: a processing module specifically used to detect the current environment of the vehicle; the processing module specifically used to determine the environmental sensitivity coefficient corresponding to the current environment from an environmental level table; wherein the environmental level table is used to indicate the effectiveness of sensors under various environments.

[0025] Optionally, the vehicle door opening warning method further includes: a processing module specifically used to determine the sensor health of the sensor based on a historical false detection rate of the sensor.

[0026] Optionally, the door opening prediction model is a spatiotemporal network model; the spatiotemporal network model includes at least: an efficient re-parameterized backbone network, a spatiotemporal attention module, an optical flow trajectory tracking module and an LSTM prediction module.

[0027] Optionally, the vehicle door opening warning method also includes: a processing module for sending vehicle-to-everything (V2X) communication information to the target vehicle; the V2X communication information is used to indicate the door opening status; the V2X communication information includes at least one of the following: door identification, door opening angle, data confidence, and timestamp.

[0028] In a third aspect, a controller is also provided, characterized in that the controller includes: a processor and a memory; the memory stores instructions executable by the processor; when the processor is configured to execute the instructions, the controller implements the method of the first aspect mentioned above.

[0029] In a fourth aspect, the present application provides a computer-readable storage medium, which includes: computer software instructions; when the computer software instructions are executed in an electronic device, the electronic device implements the method of the first aspect above.

[0030] In a fifth aspect, the present application provides a computer program product. When the computer program product is run on a computer, it enables the computer to execute the steps of the relevant method described in the first aspect to implement the method of the first aspect.

[0031] These and other aspects of the present application will become more readily apparent from the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0033] Figure 1 A schematic diagram of an application scenario of a vehicle door opening warning method provided in an embodiment of the present application;

[0034] Figure 2 A schematic flow chart of a vehicle door opening warning method provided in an embodiment of the present application;

[0035] Figure 3This is a flow chart of another vehicle door opening warning method provided by an embodiment of the present application;

[0036] Figure 4 This is a flow chart of a dynamic weight fusion algorithm provided in an embodiment of the present application;

[0037] Figure 5 This is a schematic diagram of a door opening prediction model flow chart provided in an embodiment of the present application;

[0038] Figure 6 This is a schematic diagram of a vehicle V2X communication protocol interaction provided by an embodiment of the present application;

[0039] Figure 7 A complete flowchart of a vehicle door opening warning method provided in an embodiment of the present application;

[0040] Figure 8 A schematic diagram of the composition of a vehicle door opening warning device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0042] It should be noted that in the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being more preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0043] In order to facilitate a clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, words such as "first" and "second" are used to distinguish between identical or similar items with basically the same functions and effects. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order.

[0044] "Door-opening" accidents have attracted considerable attention due to their severity. However, current mainstream early warning technologies have significant shortcomings: They lack accurate perception of rapidly approaching non-motor vehicles or pedestrians from behind (susceptible to environmental interference and blind spots), and struggle to accurately predict the driver's or occupants' true intention to open the door. Furthermore, their early warning methods are overly simplistic, lacking effective warnings and proactive intervention measures for potential victims outside the vehicle. This combination of "imprecise perception" and "simplistic processing" severely limits early warning effectiveness, necessitating the integration of multiple technologies for more precise and intelligent proactive protection.

[0045] Against this technical background, an embodiment of the present application provides a vehicle door opening warning method and controller. The method includes: acquiring multimodal data; the multimodal data includes first modal data, second modal data, third modal data, and fourth modal data; wherein the first modal data is used to indicate changes in the vehicle's topological structure; the second modal data is used to indicate changes in radial velocity when the door is opened; the third modal data is used to indicate a two-dimensional image associated with the door opening; and the fourth modal data is used to indicate the distance between the obstacle and the door; inputting the multimodal data into a door opening prediction model to obtain an output result; determining different warning levels based on the output result, and adopting different warning strategies at different warning levels. It can be seen that the embodiment of the present application inputs the multimodal data into the door opening prediction model, determines different warning levels based on the output result, and then adopts different warning strategies. On the one hand, the complementary multimodal data breaks through the bottleneck of single data, thereby improving the accuracy of door opening detection in complex scenarios. On the other hand, different warning stages are determined, and different warning strategies are then adopted for warning, further reducing the possibility of "door opening kill" occurring. Based on the above two aspects, it is possible to accurately warn of the door opening situation and reduce the risk of "door opening kill".

[0046] Figure 1 Schematic diagram of an application scenario of a vehicle door opening warning method provided in an embodiment of the present application. The vehicle door opening warning method provided in this application can be applied to Figure 1 In the application scenario shown.

[0047] like Figure 1 As shown, the application scenario includes a controller 101, a hybrid solid-state lidar sensor 102, a 77GHz millimeter wave radar sensor 103, a global shutter camera acquisition 104 and a high-frequency ultrasonic sensor 105.

[0048] Among them, the controller 101 is responsible for monitoring sensor 101 data, processing information, making decisions and executing commands to accurately control the various subsystems of the vehicle to ensure its safe, efficient and comfortable operation.

[0049] The hybrid solid-state LiDAR sensor 102 is a LiDAR technology with a fixed core optical component but uses small, micro-mechanical components to achieve beam scanning. It combines the large field of view advantages of mechanically rotating radar with the high reliability and low-cost potential of pure solid-state radar. It is used to acquire data on changes in the vehicle's topology.

[0050] The 77GHz millimeter-wave radar sensor 103 is a core sensor operating in the 77GHz frequency band, enabling the vehicle to accurately detect distance, speed, and direction in all weather conditions. It is used to acquire data indicating changes in radial velocity when a vehicle door is opened.

[0051] The global shutter camera capture 104 is an image sensor operating mode whose core feature is that all pixels on the image sensor collect light at the same instant (same exposure time). The global shutter camera 104 is used to capture a two-dimensional image associated with the door opening.

[0052] The high-frequency ultrasonic sensor 105 is a sensor that uses sound waves above the human hearing range for detection, distance measurement, imaging, or identification. The high-frequency ultrasonic sensor 105 is used to obtain data related to the distance between the obstacle and the vehicle door.

[0053] It can be understood that a vehicle door opening warning method provided in an embodiment of the present application is to output the multimodal data obtained by the hybrid solid-state lidar sensor 102, the 77GHz millimeter wave radar sensor 103, the global shutter camera 104 and the high-frequency ultrasonic sensor 105 to the controller 101, and the controller 101 processes the multimodal data and inputs the multimodal data into the door opening prediction model. Different warning levels are determined according to the output results, and different warning strategies are then adopted. This can accurately warn of the door opening situation and reduce the risk of "door opening killing".

[0054] Figure 2 A flow chart of a vehicle door opening warning method provided in an embodiment of the present application, which can be applied to Figure 1 In the application scenario shown, the following steps S201-S203 are specifically included:

[0055] S201: Acquire multimodal data.

[0056] Among them, the multimodal data includes first modal data, second modal data, third modal data and fourth modal data; among them, the first modal data is used to indicate the change in the topological structure of the vehicle; the second modal data is used to indicate the change in radial velocity when the door is opened; the third modal data is used to indicate the two-dimensional image associated with the opening of the door; and the fourth modal data is used to indicate the distance between the obstacle object and the door.

[0057] The following describes each modal data separately.

[0058] The first modal data may be acquired through a hybrid solid-state lidar sensor.

[0059] For example, a hybrid solid-state LiDAR can be positioned in the center of the vehicle's roof, collecting a 3D point cloud of the door area to detect density changes in the edge point cloud. A sudden change in density can be used to determine a topological change when the door is opened. A 3D point cloud is a collection of numerous points that specifically describes the surface geometry and spatial position of the vehicle door and its surrounding area. Each point cloud has specific 3D coordinates, providing precise 3D coordinates for door opening detection.

[0060] It should be noted that the use of hybrid solid-state lidar can make up for the shortcomings of pure vision solutions lacking depth information and millimeter-wave radar lacking structural information.

[0061] The second modal data can be obtained by a 77GHz millimeter-wave radar sensor. The 77GHz millimeter-wave radar is a core sensor operating in the 77GHz frequency band that enables vehicles to accurately detect distance, speed, and direction in all weather conditions.

[0062] The radial velocity change when the door is opened refers to the speed of the outermost edge of the door when the door is opened. For example, for safety speed reference, the speed of the outermost edge of the door is recommended to be less than or equal to 0.5 m / s.

[0063] For example, 77GHz millimeter-wave radar sensors can be deployed at the four corners of the vehicle to detect sudden changes in radial velocity when a door is opened, distinguishing door movement from vehicle body sway. For example, a radial velocity greater than 0.5 m / s is considered abnormal movement.

[0064] It should be noted that the use of 77GHz millimeter-wave radar can complement the sensitivity of hybrid solid-state lidar to small speed changes.

[0065] The third modality data may be acquired through a global shutter camera.

[0066] For example, a global shutter camera can be installed on the vehicle's exterior rearview mirror or on the vehicle's roof. By capturing the movement trajectory of the reflective stripes on the door and the passenger's body movements (such as the hand touching the door handle), it obtains a two-dimensional image associated with the opening of the door and provides two-dimensional visual features.

[0067] It should be noted that the use of a global shutter camera can assist in verifying the intention to open the door in scenes with sufficient lighting.

[0068] The fourth modal data may be acquired through a high-frequency ultrasonic sensor.

[0069] For example, a high-frequency ultrasonic sensor can be set at the edge of the vehicle door to detect obstacles (such as pedestrian legs and road shoulders) at close range. When the distance is less than 1.5 meters, an obstacle presence signal is triggered, thereby collecting data on the distance between the obstacle object and the vehicle door.

[0070] It should be noted that the use of high-frequency ultrasonic sensors can make up for the blind spot defect of hybrid solid-state lidar at close distances of less than 1 meter from the vehicle.

[0071] It can be understood that the hybrid solid-state lidar sensor and the 77GHz millimeter-wave radar sensor are jointly constrained by the changes in the vehicle's topological structure and radial velocity respectively, and can accurately detect the tiny deformation at the initial stage of door opening. The cross-validation of the two-dimensional image associated with the door opening obtained by the global shutter camera and the data collection of the distance between the obstacle object and the door by the high-frequency ultrasonic sensor improves the accuracy of complex scene detection.

[0072] S202: Input the multimodal data into the door opening prediction model to obtain an output result.

[0073] In the embodiment of the present application, after acquiring the multimodal data, the controller may input the multimodal data into the vehicle door opening prediction model to obtain an output result. The output result may reflect the opening status of the door.

[0074] In some embodiments, the output result may include: data confidence, door opening angle, and obstacle distance. The data confidence indicates the data credibility of the current output result. If the data confidence is less than a certain threshold (e.g., less than 85%), it indicates that the output result of the current detection cycle is not accurate enough, and the controller may ignore the output result. If the data confidence is greater than a certain threshold, it indicates that the output result is relatively accurate, and the process continues with S203.

[0075] In some embodiments, as Figure 3 As shown, the above S202 can be specifically implemented as follows:

[0076] S2021. Determine the weight of each sensor based on the environmental sensitivity coefficient and sensor health of the sensor corresponding to each modal data.

[0077] In some embodiments, determining the weight of each sensor based on the environmental sensitivity coefficient and sensor health of each sensor corresponding to each modal data includes: determining the weight of the sensor using the following expression:

[0078]

[0079] Among them, W i (t) represents the weight of the sensor, α i (t) represents the environmental sensitivity coefficient of the sensor, β i (t) represents the sensor health of the sensor. j (t) represents the environmental sensitivity coefficient of any sensor, β i (t) represents the sensor health of any sensor.

[0080] The following describes how to obtain the environmental sensitivity coefficient and sensor health.

[0081] In some embodiments, the current environment of the vehicle is detected; and an environmental sensitivity coefficient corresponding to the current environment is determined from an environmental level table; wherein the environmental level table is used to indicate the effectiveness of sensors in various environments.

[0082] Among them, the environmental sensitivity coefficient is obtained by looking up the environmental level table based on the light / rain and fog level, and the value range is 0.1-1. The level table is customized based on multimodal historical statistical data and is used to dynamically adjust the reliability of the sensor in different environments.

[0083] For example, on a sunny day, the visual environment sensitivity coefficient corresponding to the third modal data is 0.9, and the lidar environment sensitivity coefficient corresponding to the first modal data is 0.8; on a rainy night, the visual environment sensitivity coefficient corresponding to the third modal data is 0.3, and the lidar environment sensitivity coefficient corresponding to the first modal data is 0.7.

[0084] In some embodiments, the sensor health of a sensor is determined based on a historical false detection rate of the sensor.

[0085] Sensor health is dynamically calculated based on the historical false detection rate within a sliding window. The value range is 0.5-1, reflecting the sensor's current operating status. This historical false detection rate is calculated by dynamically maintaining a fixed time / sample window of detection results and calculating the false detection rate within that window in real time.

[0086] For example, if a millimeter-wave radar sensor has five consecutive false detections, the sensor health level will drop from 0.9 to 0.6.

[0087] S2022. According to the weight of each sensor, the multimodal data is input into the door opening prediction model with the corresponding weight to obtain an output result.

[0088] It's understood that the sensor weight calculation method expressed in the above expression utilizes a dynamic weight fusion algorithm. The weight of a sensor is calculated by dividing the product of its environmental sensitivity coefficient and its sensor health by the sum of the products of the environmental sensitivity coefficients and their sensor health for all sensors. This is used to determine the weight of the current sensor. A larger weight indicates that the sensor is less affected by external factors in the current scenario, and the collected data is more accurate. A smaller weight indicates that the sensor is more susceptible to external interference in the current scenario, so its weight is reduced to prevent it from providing interfering data and affecting subsequent data analysis.

[0089] For example, in a rainy night scene, the weight of the hybrid solid-state lidar is 0.7, which dominates the collection of the first modality data. The weight of the global shutter camera is 0.2, which assists in the collection of the third modality data, thereby improving the reliability of the data in complex environments.

[0090] For example, in backlit scenes, the global shutter camera is reduced from 0.9 to 0.3, and the weight of the hybrid solid-state lidar is increased from 0.8 to 0.9. Combined with the point cloud denoising algorithm, the accuracy of backlit detection is improved.

[0091] For example, in a sensor failure scenario, when the false detection rate of a millimeter-wave radar is greater than 20%, the system automatically increases the weight of the hybrid solid-state lidar and high-frequency ultrasonic sensor to ensure accuracy in redundant mode.

[0092] Figure 4 This is a flow chart of a dynamic weight fusion algorithm provided by an embodiment of the present application. It is understandable that, if Figure 4 As shown, based on multimodal data, environmental status and sensor health assessments are performed. The environmental sensitivity coefficient is obtained by looking up the environmental level table, and the sensor health is dynamically determined by using a sliding window to calculate historical false detection rates. These two indicators determine the weights of each sensor. A dynamic weight fusion algorithm adapts to environmental changes in real time, fusing the complementary features of multi-source data to output multimodal fusion features. In other words, the dynamic weight fusion algorithm adapts to environmental changes, dynamically assigning weights based on the environmental sensitivity coefficient and sensor health, effectively suppressing single sensor defects and achieving a detection accuracy greater than or equal to 95% in complex scenarios. It also provides high backlight detection accuracy, ensuring accuracy even in redundant mode.

[0093] In some embodiments, the door opening prediction model is a spatiotemporal network model, which includes at least: an EfficientRep backbone network, a spatiotemporal attention module, an optical flow trajectory tracking module, and an LSTM prediction module.

[0094] In some embodiments, the EfficientRep backbone network is based on a lightweight architecture called the re-parameterized visual geometry group network (RepVGG), which extracts shallow features of the door edge and reduces computational overhead by 30% compared to the single-stage real-time object detection algorithm (you only look once version 5, YOLOv5). RepVGG uses complex branches for training to improve performance and merges them into a single 3x3 convolution during inference for high-speed deployment. YOLOv5 is a fast, accurate, and easy-to-use real-time object detection algorithm that can simultaneously identify the identities and locations of multiple objects in a single image.

[0095] In some embodiments, the spatiotemporal attention module is divided into two-dimensional (2D) spatial attention and one-dimensional (1D) temporal attention. The 2D spatial attention is to generate an attention mask for the door edge area through convolution, for example, focusing on the door hinge point cloud to suppress interference from other parts of the car body; the 1D temporal attention is to apply temporal weights to the feature sequence of 3 consecutive frames to enhance dynamic features that change over time (such as the displacement trend of edge points). It should be noted that the spatiotemporal attention module in the embodiment of the present application focuses on the door edge (such as the hinge point cloud) through 2D spatial attention, and combines 1D temporal attention to track the trajectory of 3 consecutive frames to suppress interference from car body shaking.

[0096] In some embodiments, optical flow trajectory tracking is to calculate pixel-level motion vectors for the door candidate region generated by a reparameterization path aggregation network (Rep-PAN) to construct a temporal trajectory matrix.

[0097] In some implementations, the LSTM angle prediction layer takes as input a trajectory matrix and predicts the opening angle via a multi-dimensional hidden layer. It should be noted that while related technologies can only passively identify existing states, the present embodiment incorporates a door edge point motion vector sequence modeling into the LSTM angle prediction layer, enabling early prediction of door opening trends. For example, if an angle increase rate exceeding 5° per millisecond is detected for two consecutive frames, a "rapid opening" prediction is detected and an early warning is triggered, shortening response time.

[0098] Figure 5 This is a schematic diagram of a door opening prediction model flow chart provided by an embodiment of the present application. Figure 5As shown, the door opening prediction model's workflow is as follows: Multimodal fusion features are acquired, shallow features of the door edge are extracted using the EfficientRep backbone network, and candidate door regions are generated using the Rep-PAN feature pyramid. The spatiotemporal attention module then suppresses interference from other vehicle body parts using a 2D spatial attention module and applies temporal weights to the feature sequence of three consecutive frames using a 1D temporal attention module, enhancing dynamic features that change over time. Optical flow trajectory tracking is then used to calculate pixel-level motion vectors and construct a temporal trajectory matrix. Finally, the LSTM angle prediction layer uses multidimensional hidden layers to predict the door opening angle and confidence level. This door opening prediction model achieves deep analysis of dynamic features and suppresses interference from vehicle body sway.

[0099] S203. Determine different warning levels according to the output results, and adopt different warning strategies at different warning levels.

[0100] In some embodiments, adopting different warning strategies at different warning levels includes: at a first warning level, prompting the display screen to open the vehicle door.

[0101] For example, the first warning level indicates a low-risk scenario, requiring the driver to be alerted and a door open notification displayed on the display. For example, at the first warning level, the head-up display (HUD) will illuminate with a yellow warning, while the instrument cluster will display a text message stating "right front door open." The response time is less than 100 milliseconds.

[0102] At the second warning level, a buzzer will sound an alarm and the maximum angle of the door opening will be limited.

[0103] For example, the second warning level represents a medium-risk scenario, where physical restrictions on the door opening angle are implemented to prevent collisions. This includes a buzzer alarm and a maximum door opening angle limit. For example, at the second warning level, the buzzer automatically sounds an alarm, and the electric door lock is limited to an angle of less than 30°. The response time is less than 150 milliseconds.

[0104] At the third warning level, the vehicle doors are locked and an emergency stop signal is broadcast; the emergency stop information is used to instruct the target vehicle to activate the automatic emergency braking system (AEB).

[0105] The target vehicle is any vehicle in close proximity to the vehicle. If the system determines a collision is imminent and the driver fails to react in time, the automatic emergency braking system will proactively trigger emergency braking, thereby avoiding a collision at low speeds or significantly reducing the impact speed at medium and high speeds, minimizing casualties.

[0106] For example, the third warning level represents a high-risk scenario, requiring physical intervention and coordinated braking by surrounding vehicles, such as locking the vehicle doors and broadcasting an emergency stop signal. For example, at the third warning level, the vehicle doors are directly locked, and vehicle information is broadcast to surrounding vehicles, prompting them to activate the automatic emergency braking system.

[0107] In some embodiments, the output results include: the door opening angle and the distance to the obstacle object; when the door opening angle is greater than a first angle threshold and less than a second angle threshold, and the distance to the obstacle object is greater than the first distance threshold, it is determined to be in the first warning level; when the door opening angle is greater than the first angle threshold, and the distance to the obstacle object is greater than the second distance threshold and less than the first distance threshold, it is determined to be in the second warning level; when the door opening angle is greater than a third angle threshold, and the distance to the obstacle object is less than the second distance threshold, it is determined to be in the third warning level; wherein the third angle threshold is less than the first angle threshold.

[0108] For example, if the door opening angle in the output results is greater than 15° and less than 30°, and the obstacle distance is greater than 1.5 meters, the warning level 1 is determined. 15° is the first angle threshold, 30° is the second angle threshold, and 1.5 meters is the first distance threshold. If the obstacle distance is greater than 1.5 meters, it indicates that the obstacle is not present.

[0109] For example, if the door opening angle in the output results is greater than 15° and the obstacle distance is greater than 0.8 meters and less than 1.5 meters, the second warning level is determined, where 15° is the first angle threshold, 0.8 meters is the second distance threshold, and 1.5 meters is the first distance threshold. If the obstacle distance is greater than 0.8 meters and less than 1.5 meters, it indicates the presence of an obstacle.

[0110] For example, if the door opening angle in the output result is greater than 10° and the obstacle distance is less than 0.8 meters, the third warning level is determined, where 10° is the third angle threshold and 0.8 meters is the second distance threshold. An obstacle distance less than 0.8 meters indicates an imminent danger.

[0111] It can be understood that the vehicle door opening warning method provided in the embodiment of the present application mainly determines different warning levels based on the door opening angles and obstacle object distances in different situations. Based on different warning levels, different warning signals are given and different warning measures are taken, so as to better provide warnings and effectively avoid the occurrence of "door opening kill" situations.

[0112] In some embodiments, the vehicle door opening warning method of the embodiments of the present application further includes: transmitting V2X communication information to the target vehicle; the V2X communication information is used to indicate whether the door is open. Specifically, the V2X communication information includes at least one of the following: a door identification, a door opening angle, a data confidence level, and a timestamp.

[0113] Among them, V2X refers to the wireless communication technology between vehicles and surrounding entities (including other vehicles, infrastructure, pedestrians and cloud networks), which is used to exchange traffic, environmental and vehicle status information in real time to improve road safety and traffic efficiency.

[0114] The door ID refers to the custom field DoorID, which is used to uniquely identify the doors, such as doors 1 to 4. It should be noted that the embodiment of the present application can support independent warnings for different doors to achieve more accurate warnings.

[0115] The door opening angle refers to the custom field OpenAngle, which describes the real-time angle of the door opening, with a general accuracy of 0.1°. It should be noted that the embodiment of the present application can accurately determine the risk level by angle, thereby improving the accuracy of early warning.

[0116] The data confidence level refers to the custom field Confidence, which accurately describes the sensor's data confidence level, ranging from 0 to 100%. It should be noted that the embodiment of the present application can filter out invalid data by using the data confidence level.

[0117] The timestamp refers to the custom field Timestamp, which is used to ensure time synchronization. It should be noted that the embodiment of the present application supports predicted shutdown time to reduce device energy consumption.

[0118] In some embodiments, the custom message format supports independent warnings for each door. For example, when it is detected that the opening angle of the right front door of the vehicle in front is 25° and the obstacle distance is 1.2m, the HUD warning of the vehicle behind is triggered through V2X broadcast, and the warning range is extended to a surrounding 300m.

[0119] In some implementations, V2X communication utilizes dual-link transmission via a direct PC5 interface and a Uu cellular network, ensuring a low cyclic redundancy check (CRC) error rate. PC5 handles real-time hazards, while Uu provides global decision-making, achieving a combination of "local response + cloud intelligence."

[0120] Among them, PC5 is a direct communication interface defined by 3GPP, which enables direct communication between vehicles / devices (without base station transfer) and realizes (V2V), (V2P), Communication between devices on the vehicle-to-infrastructure (V2I) side.

[0121] Among them, Uu is the cellular communication interface defined by 3GPP, which enables vehicles / equipment to connect to the core network through base stations, realizing (V2N) communication.

[0122] It should be noted that the embodiment of the present application is PC5 direct connection (delay less than 10 milliseconds) and Uu cellular network dual link transmission, with CRC check (bit error rate less than 10 -6 ) to ensure that the reliable transmission rate of door status messages is greater than 99.9% in complex scenarios such as tunnels and densely populated urban areas, avoiding early warning failures caused by single channel failures.

[0123] For example, if the leading vehicle suddenly brakes, an emergency message is broadcast to the following vehicle via the PC5 interface. The following vehicle automatically applies the brakes within 10ms to avoid a collision. The vehicle then reports its location to the cloud-based traffic brain via the Uu interface. The cloud calculates the overall road conditions and sends a detour route to the vehicle's central control screen.

[0124] Figure 6 This is a schematic diagram of a vehicle V2X communication protocol interaction provided by an embodiment of the present application. It can be understood that, if Figure 6 As shown, in an embodiment of the present application, a local vehicle broadcasts a message through V2X, wherein the broadcast message includes a door identification, a door opening angle, data confidence, a timestamp, etc. When surrounding vehicles capture the broadcast message, the surrounding vehicles will perform data verification through sensors. If the verification is successful, a graded response will be executed. If the verification fails, the message will be ignored.

[0125] It should be noted that the tiered responses of surrounding vehicles differ from the warnings implemented by the local vehicle. If the broadcast message verification determines that the local vehicle has implemented a Level 1 or Level 2 warning, the HUD of the surrounding vehicle receiving the V2X broadcast message will display the warning information. If the broadcast message verification determines that the local vehicle has implemented a Level 3 warning, the surrounding vehicle receiving the V2X broadcast message will activate the automatic emergency braking system and broadcast the vehicle signal to other surrounding vehicles, prompting them to issue warnings. If the verification fails, the message will be ignored.

[0126] For example, when the surrounding area receives a local vehicle warning message, it will verify the local vehicle's V2X broadcast message, such as DoorID = 2 (right front door), OpenAngle = 25°, Confidence = 95%. If the verification is successful, the HUD of the surrounding vehicles that receive the V2X broadcast information will issue a corresponding warning.

[0127] Figure 7This is a complete flow chart of a vehicle door opening warning method provided in an embodiment of the present application. Figure 7 As shown, the specific steps include S701-S704.

[0128] S701. Collect data of different modalities through multiple sensors.

[0129] S702: Input the multimodal data into the dynamic weight fusion algorithm.

[0130] S703: Input the data features fused by the dynamic weight fusion algorithm into the spatiotemporal network model.

[0131] S704: Based on the output of the spatiotemporal network model, a hierarchical mechanism is adopted. Meanwhile, the local vehicle conducts V2X collaborative communication with surrounding vehicles.

[0132] The execution units adopted by the hierarchical response mechanism include but are not limited to HUD warning, buzzer alarm, electric door suction control, and AEB braking. Different execution units are selected according to different response mechanisms.

[0133] It can be understood that the embodiment of the present application collects data of different modalities through a variety of sensors, inputs the multimodal data into a dynamic weight fusion algorithm, and then inputs the data features after fusion by the dynamic weight fusion algorithm into the space-time network model. After that, a hierarchical response mechanism is performed according to the output result of the space-time network model, and different execution units are selected according to different response mechanisms. At the same time, the local vehicle and the surrounding vehicles perform V2X collaborative communication. The complementarity of multimodal data breaks through the bottleneck of single data, thereby improving the accuracy of door opening detection in complex scenarios; the door opening prediction model improves the accuracy of early warning by making timely adjustments to the vehicle's surrounding environment; determines different early warning stages, and then adopts different early warning strategies for early warning, further reducing the possibility of "door opening kill". Based on the above content, the door opening situation can be accurately warned, reducing the risk of "door opening kill".

[0134] The embodiment of the present application can divide the functional modules of the above-mentioned electronic device according to the above-mentioned method example. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0135] In some embodiments, the present application also provides a vehicle door opening warning device, which may include one or more functional modules for implementing the vehicle door opening warning method described in the above method embodiment.

[0136] For example, Figure 8 This is a schematic diagram of the composition of a vehicle door opening warning device provided in an embodiment of the present application. Figure 8 As shown, the device includes: an acquisition module 801, a processing module 802 and an early warning module 803.

[0137] The acquisition module 801 is used to acquire multimodal data; the multimodal data includes first modal data, second modal data, third modal data and fourth modal data; wherein the first modal data is used to indicate the change in the topological structure of the vehicle; the second modal data is used to indicate the change in radial velocity when the door is opened; the third modal data is used to indicate the two-dimensional image associated with the opening of the door; the fourth modal data is used to indicate the distance between the obstacle object and the door; the processing module 802 is used to input the multimodal data into the door opening prediction model to obtain an output result; the warning module 803 is used to determine different warning levels according to the output results and adopt different warning strategies at different warning levels.

[0138] Optionally, the warning module 803 is used to adopt different warning strategies at different warning levels, including: the warning module 803 is specifically used to, at the first warning level, prompt the door to be opened through the display screen; the warning module 803 is specifically used to, at the second warning level, alarm through the buzzer and limit the maximum angle of the door opening; the warning module 803 is specifically used to, at the third warning level, lock the door and broadcast an emergency stop signal; the emergency stop information is used to instruct the target vehicle to activate the automatic emergency braking system (AEB).

[0139] Optionally, the output results include: the door opening angle and the obstacle object distance; the processing module 802 is specifically used to determine that it is in the first warning level when the door opening angle is greater than the first angle threshold and less than the second angle threshold, and the obstacle object distance is greater than the first distance threshold; the processing module 802 is specifically used to determine that it is in the second warning level when the door opening angle is greater than the first angle threshold and the obstacle object distance is greater than the second distance threshold and less than the first distance threshold; the processing module 802 is specifically used to determine that it is in the third warning level when the door opening angle is greater than the third angle threshold and the obstacle object distance is less than the second distance threshold; wherein, the third angle threshold is less than the first angle threshold.

[0140] Optionally, the processing module 802 is used to input the multimodal data into the door opening prediction model to obtain an output result, including: determining the weight of each sensor based on the environmental sensitivity coefficient and sensor health of the sensor corresponding to each modal data; and inputting the multimodal data into the door opening prediction model with the corresponding weight according to the weight of each sensor to obtain the output result.

[0141] Optionally, the processing module 802 is configured to determine the weight of each sensor according to the environmental sensitivity coefficient and sensor health of the sensor corresponding to each modal data, including: using the following expression to determine the weight of the sensor;

[0142]

[0143] Among them, W i (t) represents the weight of the sensor, α i (t) represents the environmental sensitivity coefficient of the sensor, β i (t) represents the sensor health of the sensor.

[0144] Optionally, the vehicle door opening warning method also includes: the processing module 802 is specifically used to detect the current environment of the vehicle; the processing module 802 is specifically used to determine the environmental sensitivity coefficient corresponding to the current environment from the environmental level table; wherein the environmental level table is used to indicate the effectiveness of the sensor under each environment.

[0145] Optionally, the vehicle door opening warning method further includes: the processing module 802 is specifically used to determine the sensor health of the sensor according to the historical false detection rate of the sensor.

[0146] Optionally, the door opening prediction model is a spatiotemporal network model; the spatiotemporal network model includes at least: an efficient re-parameterized backbone network, a spatiotemporal attention module, an optical flow trajectory tracking module and an LSTM prediction module.

[0147] Optionally, the vehicle door opening warning method also includes: the processing module 802 is used to send vehicle-to-everything (V2X) communication information to the target vehicle; the V2X communication information is used to indicate the door opening status; the V2X communication information includes at least one of the following: door identification, door opening angle, data confidence, and timestamp.

[0148] In some embodiments, the embodiments of the present application further provide a computer-readable storage medium on which computer program instructions are stored; when the computer program instructions are executed by an electronic device, the electronic device implements the method described in the aforementioned embodiments.

[0149] In some embodiments, the embodiments of the present application further provide a computer program product, which, when executed on a computer, enables the computer to execute the above-mentioned related method steps to implement the method in the above-mentioned embodiments.

[0150] In the description of the embodiments of the present application, specific features, structures, materials or characteristics may be combined in an appropriate manner in any one or more embodiments or examples.

[0151] The above are only specific embodiments of the present application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A vehicle door opening warning method, characterized in that: The method comprises: Acquire multimodal data; the multimodal data includes first modal data, second modal data, third modal data, and fourth modal data; wherein the first modal data is used to indicate a change in the topological structure of the vehicle; the second modal data is used to indicate a change in radial velocity when the door is opened; the third modal data is used to indicate a two-dimensional image associated with the door opening; and the fourth modal data is used to indicate a distance between an obstacle and the door; Inputting the multimodal data into a door opening prediction model to obtain an output result; Different warning levels are determined according to the output results, and different warning strategies are adopted at different warning levels.

2. The method according to claim 1, characterized in that Different early warning strategies are adopted at different early warning levels, including: At the first warning level, the display screen will prompt that the door is open; At the second warning level, a buzzer will sound an alarm and the maximum angle of the door opening will be limited; At the third warning level, the vehicle doors are locked and an emergency stop signal is broadcast; the emergency stop information is used to instruct the target vehicle to activate the automatic emergency braking system (AEB).

3. The method according to claim 2, characterized in that The output results include: door opening angle and obstacle distance; When the door opening angle is greater than a first angle threshold and less than a second angle threshold, and the obstacle distance is greater than a first distance threshold, determining that the vehicle is in the first warning level; When the door opening angle is greater than a first angle threshold and the obstacle distance is greater than a second distance threshold and less than the first distance threshold, determining that the warning level is the second warning level; When the door opening angle is greater than a third angle threshold and the obstacle object distance is less than a second distance threshold, it is determined to be in the third warning level; wherein the third angle threshold is less than the first angle threshold.

4. The method according to claim 1, wherein The multimodal data is input into the door opening prediction model to obtain output results, including: Determine the weight of each sensor based on the environmental sensitivity coefficient and sensor health of the sensor corresponding to each modal data; According to the weight of each sensor, the multimodal data is input into the door opening prediction model with the corresponding weight to obtain the output result.

5. The method according to claim 4, characterized in that The step of determining the weight of each sensor according to the environmental sensitivity coefficient and sensor health of each sensor corresponding to each modal data includes: The weight of the sensor is determined using the following expression: Among them, W i (t) represents the weight of the sensor, α i (t) represents the environmental sensitivity coefficient of the sensor, β i (t) represents the sensor health of the sensor.

6. The method according to claim 4, characterized in that The method further comprises: Detect the vehicle's current environment; Determine the environmental sensitivity coefficient corresponding to the current environment from the environmental level table; wherein the environmental level table is used to indicate the effectiveness of the sensor in each environment.

7. The method according to claim 4, characterized in that The method further comprises: A sensor health of the sensor is determined based on a historical false detection rate of the sensor.

8. The method according to claim 1, characterized in that The door opening prediction model is a spatiotemporal network model; The spatiotemporal network model includes at least: an efficient reparameterized backbone network, a spatiotemporal attention module, an optical flow trajectory tracking module and a long short-term memory network LSTM prediction module.

9. The method according to claim 1, characterized in that The method further comprises: A vehicle-to-everything (V2X) communication technology (V2X) communication message is transmitted to a target vehicle; the V2X communication message is used to indicate a door opening condition; the V2X communication message includes at least one of the following: a door identification, a door opening angle, a data confidence level, and a timestamp.

10. A controller, characterized in that: The controller includes: a processor and a memory; The memory stores instructions executable by the processor; When the processor is configured to execute the instructions, the controller is caused to implement the method according to any one of claims 1 to 9.

Images