System and method for sensing vehicle emergency call based on multi-modal information fusion

Abstract

The invention provides a system and a method for sensing a vehicle emergency call based on multi-modal information fusion. The system comprises a data terminal, an SOS switch, a gateway, an intelligent cabin and a safety airbag controller, the data terminal is in communication connection with the SOS switch, the gateway, the intelligent cabin and the airbag controller, and is used for triggering an emergency call function according to a preset triggering condition and sending an emergency call message to the network equipment; the preset triggering condition comprises at least one of an action signal of a safety air bag, a CAN bus signal of vehicle collision, a voice keyword recognition triggering signal and a manual triggering signal, and multiple triggering modes are fused; according to the invention, call failure caused by failure of a single sensor or incapability of operation of a person in a real emergency is greatly avoided, and the reliability and safety of vehicle emergency call triggering are improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent vehicle control, and in particular to a vehicle emergency call system and method based on multimodal information fusion perception. Background Technology

[0002] In recent years, with the continuous development of road traffic and the increasingly widespread use of automobiles, road traffic accidents have occurred frequently, causing a large number of casualties. Many accidental personal injury injuries and deaths are due to the lack of timely rescue after the accident, resulting in serious consequences. In the event of a vehicle collision or other emergency, sending a distress signal quickly and accurately is crucial to saving the lives of drivers and passengers. In order to improve the level of accident rescue, shorten rescue time, and reduce casualties, emergency distress call technology for vehicle accidents has been researched and put into practical application both domestically and internationally.

[0003] Currently, emergency call systems are relatively limited in functionality, relying primarily on two methods: manual activation of the SOS button by the occupant, and automatic activation by the airbag control unit upon detecting a collision. However, both methods have significant drawbacks. Manual activation is impossible when the occupant is injured, unconscious, or panicked; and airbag activation has strict thresholds, meaning it cannot be activated in severe accidents that do not meet the deployment conditions (such as rollovers, slow falls, or severe flooding) or non-collision emergencies (such as sudden illness of the driver or other occupants).

[0004] To address the aforementioned problems, this invention provides a vehicle emergency call system and method based on multimodal information fusion perception, thereby resolving these issues. Summary of the Invention

[0005] In order to solve the problems existing in the prior art, this invention innovatively proposes a vehicle emergency call system and method based on multimodal information fusion perception, which effectively solves the problem of low reliability and security of vehicle emergency call triggering caused by the single triggering method in the prior art, and effectively improves the reliability and security of vehicle emergency call triggering.

[0006] The first aspect of this invention provides a vehicle emergency call system based on multimodal information fusion perception, comprising: The system comprises a data terminal, an SOS switch, a gateway, a smart cockpit, and an airbag controller. The airbag controller acquires airbag activation signals. The gateway acquires CAN bus signals indicating whether a collision has occurred. The smart cockpit acquires currently generated voice information, including preset emergency call keywords, and sends a voice keyword recognition trigger signal. The SOS switch sends a manual trigger signal for emergency calls from occupants. The data terminal is communicatively connected to the SOS switch, gateway, smart cockpit, and airbag controller, and is used to trigger the emergency call function according to preset trigger conditions and send an emergency call message to network devices. The preset trigger conditions include at least one of the following: airbag activation signals, CAN bus signals indicating a vehicle collision, voice keyword recognition trigger signals, and manual trigger signals.

[0007] Optionally, it also includes a positioning module that is communicatively connected to the data terminal; the positioning module is used to send real-time vehicle location information to the data terminal when the emergency call function is triggered.

[0008] Furthermore, it also includes a camera and an image processing unit. The camera is used to acquire real-time image information of the vehicle interior and send the image information to the image processing unit. The image processing unit is used to identify the emergency status of the occupants of the vehicle based on the current image information of the vehicle interior, and when an emergency status of the occupants of the vehicle interior is identified, it sends an image emergency status recognition trigger signal to the data terminal. The emergency status includes: the driver is unconscious or unresponsive, or there is smoke or open flame inside the vehicle. The preset trigger condition also includes the image emergency status recognition trigger signal.

[0009] Furthermore, the emergency call message includes: the vehicle's real-time location information, vehicle identification number, reason for triggering the emergency call, type of call triggered, and on-site images or pictures of the vehicle's interior captured by the camera.

[0010] The second aspect of this invention provides a vehicle emergency call method based on multimodal information fusion perception, implemented on the basis of a vehicle emergency call system based on multimodal information fusion perception provided in the first aspect of this invention, comprising: The airbag controller receives the activation signal of the airbag; The gateway obtains the CAN bus signal indicating whether a vehicle collision has occurred; The intelligent cockpit acquires currently generated voice information, which includes preset emergency call keywords, and sends a voice keyword recognition trigger signal. The SOS switch sends a manual trigger signal when an emergency call is made by someone inside the vehicle. The data terminal triggers the emergency call function according to preset trigger conditions and sends an emergency call message to the network device; the preset trigger conditions include at least one of the following: airbag activation signal, CAN bus signal of vehicle collision, voice keyword recognition trigger signal, and manual trigger signal.

[0011] Optionally, it also includes: When the emergency call function is triggered, a response message is sent to the vehicle body controller, the doors are unlocked, and the hazard lights and right turn alarm are activated.

[0012] Furthermore, it also includes: The camera acquires real-time images of the vehicle's interior and sends the images to the image processing unit. Based on the images, the image processing unit identifies the emergency status of the occupants and sends an image emergency status recognition trigger signal to the data terminal when an emergency status is detected. The emergency status includes: the driver being unconscious or unresponsive, or the presence of smoke or open flame inside the vehicle. The preset triggering condition also includes the image emergency status recognition trigger signal.

[0013] Furthermore, based on the current image information inside the vehicle, identifying the emergency status of the occupants specifically includes: Acquire the vehicle's CAN bus data, which includes at least the steering wheel grip force status; Parallel analysis of emergency states for vehicle occupants based on computer vision algorithm models using video stream data and CAN bus data; specifically, when the emergency state is that the driver is unconscious or unresponsive, the parallel analysis of emergency states for vehicle occupants based on computer vision algorithm models using video stream data and CAN bus data includes: Based on the detected facial feature points, the duration of the driver's eye closure is calculated, and the driver's unconscious or unresponsive state is identified based on the duration of the driver's eye closure and the steering wheel grip strength. When the duration of the driver's eye closure exceeds a preset threshold and the steering wheel grip strength is characterized by the disappearance of grip strength, the driver is determined to be in an unconscious or unresponsive state.

[0014] Furthermore, when the emergency situation involves smoke or open flame inside the vehicle, the parallel analysis of the emergency situation of the occupants inside the vehicle based on the computer vision algorithm model, using video stream data and CAN bus data, specifically includes: The system identifies whether there are visual features in the video stream that correspond to smoke or open flame. When there are visual features in the video stream that correspond to smoke or open flame, it determines that there is smoke or open flame inside the vehicle. Visual features include color features and texture features. Optionally, the emergency state further includes: the occupants of the vehicle are in a state of distress; when the emergency state is that the occupants of the vehicle are in a state of distress, the parallel analysis of the emergency state of the occupants of the vehicle based on the computer vision algorithm model on the video stream data and CAN bus data specifically includes: By estimating the posture, the skeletal key point data of the occupants in the vehicle is obtained, and the skeletal key point data of the occupants in the vehicle is matched with a preset abnormal posture model. If the matching degree exceeds a preset percentage threshold, the occupants in the vehicle are determined to be in a state of pain.

[0015] The technical solution adopted in this invention has the following technical effects: 1. In the technical solution of this invention, the data terminal is communicatively connected to the SOS switch, gateway, smart cockpit, and airbag controller, respectively, and is used to trigger the emergency call function according to preset trigger conditions and send emergency call messages to network devices. The preset trigger conditions include at least one of the following: airbag action signal, CAN bus signal of vehicle collision, voice keyword recognition trigger signal, and manual trigger signal. It integrates multiple methods such as manual trigger, collision signal trigger, voice trigger, and airbag action trigger, which greatly avoids call failure due to the failure of a single sensor or the inability of personnel to operate in real emergency situations. It effectively solves the problem of low reliability and safety of vehicle emergency call triggering caused by the single vehicle emergency call triggering method in the prior art, and effectively improves the reliability and safety of vehicle emergency call triggering.

[0016] 2. In the technical solution of the present invention, the image processing unit can also identify the emergency status of the occupants inside the vehicle based on the current image information inside the vehicle, and send an image emergency status recognition trigger signal to the data terminal when the emergency status of the occupants inside the vehicle is identified. The emergency status includes: the driver is unconscious or unresponsive, smoke or open flame is present inside the vehicle (and may also include the occupants being in pain). The preset trigger condition also includes the image emergency status recognition trigger signal, which can more accurately identify real emergency situations, effectively reduce the false alarm rate of non-emergency events, save social rescue resources, and further improve the reliability and safety of vehicle emergency call triggering.

[0017] 3. The emergency call message in the technical solution of this invention includes: the vehicle's real-time location information, vehicle identification code, the reason for triggering the emergency call, the type of call triggered, and on-site images or pictures of the vehicle's interior collected from the camera, enabling the rescue center to understand the on-site situation of the vehicle and the condition of the people inside the vehicle in the first instance, providing key information support for formulating accurate rescue plans.

[0018] 4. In the technical solution of this invention, when the emergency call function is triggered, a response message is sent to the vehicle body controller to unlock the doors and activate the hazard lights and right turn alarm; this alerts passing vehicles and prevents secondary traffic accidents.

[0019] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of the system structure in Embodiment 1 of the present invention; Figure 2 This is a flowchart illustrating the method of Embodiment 1 in the present invention; Figure 3 This is a schematic diagram of the process of identifying an emergency state through in-vehicle image information in Embodiment 1 of the present invention. Detailed Implementation

[0022] To clearly illustrate the technical features of this solution, the invention will be described in detail below through specific embodiments and in conjunction with the accompanying drawings. The following disclosure provides many different embodiments or examples for implementing different structures of the invention. To simplify the disclosure of the invention, components and arrangements of specific examples are described below. Furthermore, reference numerals and / or letters may be repeated in different examples. This repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed. It should be noted that the components illustrated in the drawings are not necessarily drawn to scale. Descriptions of well-known components, processing techniques, and processes are omitted in this invention to avoid unnecessarily limiting the invention.

[0023] Example 1 like Figure 1 As shown, the present invention provides a vehicle emergency call system based on multimodal information fusion perception, comprising: The system comprises a data terminal, an SOS switch, a gateway, a smart cockpit, and an airbag controller. The airbag controller acquires airbag activation signals. The gateway acquires CAN bus signals indicating whether a collision has occurred. The smart cockpit acquires currently generated voice information, including preset emergency call keywords, and sends a voice keyword recognition trigger signal. The SOS switch sends a manual trigger signal for emergency calls from occupants. The data terminal is communicatively connected to the SOS switch, gateway, smart cockpit, and airbag controller, and is used to trigger the emergency call function according to preset trigger conditions and send emergency call messages to network devices. The preset trigger conditions include at least one of the following: airbag activation signals, CAN bus signals indicating a vehicle collision, voice keyword recognition trigger signals, and manual trigger signals.

[0024] As the core processing and decision-making unit of the system, the data terminal is used to receive, process and analyze data from various components, and to execute logical judgments and controls for emergency calls.

[0025] The SOS switch can be placed in a prominent location inside the vehicle to receive manual emergency call commands from users.

[0026] Gateway: Used to enable data exchange and protocol conversion between various electronic control units within the system, especially to obtain relevant vehicle status information from the vehicle's CAN bus.

[0027] Intelligent cockpit: It integrates a voice recognition module to collect and process the voice data of people in the vehicle and can recognize preset emergency call keywords.

[0028] Airbag controller: Used to monitor vehicle collision events and output airbag deployment signals when a collision occurs.

[0029] The data terminal is connected to the SOS switch, camera, mobile communication unit, gateway, smart cockpit and airbag controller, and is configured to automatically trigger the emergency call function according to at least one of the preset trigger conditions.

[0030] Triggering conditions may include: 1. Receive an action signal from the airbag controller.

[0031] 2. The vehicle CAN bus data obtained through the gateway meets the preset collision conditions, which are determined based on at least one parameter among vehicle deceleration, tilt angle, and pressure change.

[0032] 3. The smart cockpit's voice recognition module identifies preset emergency call keywords.

[0033] Specifically, the emergency call function is triggered based on preset trigger conditions as follows: (1) If the SOS switch is detected to be manually triggered, the emergency call function is triggered.

[0034] (2) If an airbag activation signal is detected, the emergency call function is triggered.

[0035] (3) If the vehicle CAN bus data obtained through the gateway indicates that the vehicle has been involved in a serious collision, the emergency call function will be triggered.

[0036] (4) If the smart cockpit recognizes the preset emergency call keywords and sends a voice keyword recognition trigger signal, the emergency call function will be triggered.

[0037] Specifically, the triggering priorities among the airbag activation signal, the CAN bus signal of a vehicle collision, the voice keyword recognition trigger signal, and the manual trigger signal, in descending order, can be: airbag activation signal, CAN bus signal of a vehicle collision (the priority of the airbag activation signal can also be the same as the priority of the CAN bus signal of a vehicle collision), voice keyword recognition trigger signal, and manual trigger signal.

[0038] One implementation method is as follows: First, obtain the highest priority trigger signal. If a high-priority trigger signal has been received, the emergency call function is triggered directly, without considering lower-priority trigger signals. If no high-priority trigger signal has been received, other trigger signals are obtained sequentially in descending order of priority (e.g., Figure 2 (as shown) Another implementation method is to determine the risk level of the emergency call function based on the number of trigger signals received. The more trigger signals received, the higher the risk level of the emergency call function, the higher the corresponding response level, and the shorter the response time.

[0039] Specifically, the smart cockpit can also be equipped with a microphone and a speaker. The microphone can be used to acquire the voice information of the people inside the vehicle and send the voice information to the voice recognition module to recognize the semantics. The speaker can be used to play the semantic information in the data terminal. The preset emergency call keywords can be keywords that are not easy for people in the vehicle to say in daily life. They can be special words set by the user, such as "red alert" or "open sesame". In daily use, in order to avoid accidental triggering, a reconfirmation mechanism (such as voice confirmation or button confirmation) can be added when the preset emergency call keywords are recognized.

[0040] The vehicle emergency call system based on multimodal information fusion also includes a positioning module (not shown in the figure) that is connected to the data terminal for communication. The positioning module obtains the current location information of the vehicle in real time and sends the real-time location information of the vehicle to the data terminal when the emergency call function is triggered.

[0041] Furthermore, the vehicle emergency call system based on multimodal information fusion perception also includes a camera and an image processing unit. The camera is used to acquire real-time image information of the vehicle interior and send the image information to the image processing unit. The image processing unit is used to identify the emergency status of the occupants of the vehicle based on the current image information of the vehicle interior, and when the emergency status of the occupants of the vehicle interior is identified, it sends an image emergency status recognition trigger signal to the data terminal. The emergency status includes: the driver is unconscious or unresponsive, or there is smoke or open flame inside the vehicle. The preset trigger condition also includes the image emergency status recognition trigger signal.

[0042] Specifically, the trigger priorities among the airbag activation signal, the CAN bus signal of a vehicle collision, the voice keyword recognition trigger signal, the manual trigger signal, and the image emergency state recognition trigger signal, in descending order, can be as follows: airbag activation signal, CAN bus signal of a vehicle collision (the priority of the airbag activation signal can also be the same as the priority of the CAN bus signal of a vehicle collision), voice keyword recognition trigger signal, manual trigger signal, and image emergency state recognition trigger signal.

[0043] Specifically, the camera includes at least one camera facing the driver's cab inside the vehicle, configured to operate in a low-power mode for acquiring image data of the occupants inside the vehicle.

[0044] Mobile communication unit: used to establish a voice and / or data communication link with remote network equipment (such as call center, cloud server) when an emergency call is triggered.

[0045] The image processing unit can be a separate image processor or an image processing module integrated into the data terminal. It analyzes and identifies preset emergency states through image data collected by the camera. The emergency states include, but are not limited to: the driver being unconscious or unresponsive, or the presence of smoke or open flame inside the vehicle.

[0046] Preferably, the system also includes a positioning module and vehicle status sensors (not shown in the figure) connected to the data terminal. The emergency call message includes at least vehicle location information, and may also include a vehicle identification number, triggering reason, triggering method type, and on-site images or short video clips captured by a camera. The sent emergency call message not only includes location and vehicle information, but may also include the triggering reason and on-site images, enabling the rescue center to understand the severity of the accident and the condition of the occupants in the vehicle as soon as possible, providing crucial information support for developing a precise rescue plan; when occupants lose the ability to actively call for help, the system can act as a "silent guardian," automatically and promptly initiating a distress call, buying precious "golden time" to save lives.

[0047] Furthermore, the image processing unit performs the following lightweight computer vision algorithm model analysis in parallel: Analysis of driver's unconscious or unresponsive state: Feature point detection is performed on the driver's face to calculate the duration of eye closure; at the same time, information such as steering wheel grip strength is obtained through the vehicle's CAN bus; when the duration of the driver's eye closure exceeds a preset time threshold (e.g., 3 seconds) and the steering wheel grip strength is "disappeared", the driver is determined to be unconscious.

[0048] The information such as steering wheel grip force status can be obtained through the vehicle CAN bus. This can be achieved by setting a grip force sensor (flexible thin film pressure sensor or distributed strain gauge) on the surface of the vehicle steering wheel and integrating the information collected by the grip force sensor into the original CAN bus network through a microcontroller. Alternatively, the torque signal of the electronic power steering system can be used to indirectly and continuously determine whether the driver is operating the steering wheel. This embodiment of the invention does not impose any limitations on this method.

[0049] Smoke or open flame detection and analysis inside the vehicle: The video stream is analyzed to identify whether there are visual features (such as specific colors, textures and dynamic diffusion patterns) corresponding to smoke or open flame. If visual features (such as specific colors, textures and dynamic diffusion patterns) corresponding to smoke or open flame exist, the environment inside the vehicle is determined to be abnormal and smoke or open flame is present.

[0050] Preferably, the emergency state may also include: the occupants of the vehicle experiencing pain; identification and analysis of the occupants experiencing pain: performing posture estimation on the occupants of the vehicle (applicable to situations with good lighting conditions, no obstructions, or multiple cameras at different angles), and obtaining their skeletal key points; matching the temporal data of the skeletal key points with a preset abnormal posture model (such as curling up, violently struggling), and if the matching degree exceeds a preset percentage threshold (e.g., 80%-90%, which can be customized), then it is determined that the occupant is in a state of pain.

[0051] It should be noted that in this embodiment, the computer vision algorithm model, driver facial feature point detection model, skeletal key point acquisition model, skeletal key point matching model, and smoke or open flame state detection model can all be obtained by annotation and training based on historical image data. Moreover, the image data used for training or transmission is obtained with the user's consent and can be analyzed based on abstract facial feature points or skeletal data. There is no need to upload or store the original video images (images are only uploaded with the user's consent in emergency situations, i.e., the in-vehicle scene images or pictures captured by the camera included in the emergency call message), thus protecting the privacy of the driver inside the vehicle.

[0052] In this invention, the data terminal is communicatively connected to the SOS switch, gateway, smart cockpit, and airbag controller, respectively, and is used to trigger the emergency call function according to preset trigger conditions and send emergency call messages to network devices. The preset trigger conditions include at least one of the following: airbag action signal, CAN bus signal indicating whether a vehicle collision has occurred, voice keyword recognition trigger signal, and manual trigger signal. This integrates multiple triggering methods, including manual triggering, collision signal triggering, voice triggering, and airbag action triggering, greatly avoiding call failures due to single sensor failure or inability of personnel to operate in real emergency situations. It effectively solves the problem of low reliability and safety of vehicle emergency call triggering caused by the single triggering method in the prior art, and effectively improves the reliability and safety of vehicle emergency call triggering.

[0053] In the technical solution of this invention, the data terminal can also identify the emergency status of the occupants inside the vehicle based on the current in-vehicle image information, and send an image emergency status recognition trigger signal when the emergency status of the occupants is identified. The emergency status includes: the driver is unconscious or unresponsive, smoke or open flame is present inside the vehicle (and may also include the occupants being in pain). The preset triggering condition also includes the image emergency status recognition trigger signal, which can more accurately identify real emergency situations, effectively reduce the false alarm rate of non-emergency events, save social rescue resources, and further improve the reliability and safety of vehicle emergency call triggering.

[0054] The emergency call message in the technical solution of this invention includes: the vehicle's real-time location information, vehicle identification code, the reason for triggering the emergency call, the type of call triggered, and on-site images or pictures of the vehicle's interior captured by the camera. This enables the rescue center to understand the on-site situation of the vehicle and the condition of the people inside the vehicle in the first instance, providing key information support for formulating a precise rescue plan.

[0055] In the technical solution of this invention, when the emergency call function is triggered, a response message is simultaneously sent to the vehicle body controller to unlock the doors and activate the hazard lights and right turn alarm; this alerts passing vehicles and prevents secondary traffic accidents.

[0056] Example 2 like Figure 2 As shown, the second aspect of the present invention provides a vehicle emergency call method based on multimodal information fusion perception, which is implemented based on the vehicle emergency call system based on multimodal information fusion perception provided in the first aspect of the present invention, and includes: Continuous monitoring of multi-source trigger signals: the airbag controller acquires the airbag's activation signal; the gateway acquires the CAN bus signal indicating whether a collision has occurred; the smart cockpit acquires currently generated voice information, including preset emergency call keywords, and sends a voice keyword recognition trigger signal; the SOS switch sends a manual trigger signal when occupants make an emergency call. The data terminal triggers the emergency call function according to preset trigger conditions and sends an emergency call message to the network device; the preset trigger conditions include at least one of the following: airbag activation signal, CAN bus signal of vehicle collision, voice keyword recognition trigger signal, and manual trigger signal.

[0057] Specifically, the emergency call function is triggered based on preset trigger conditions as follows: (1) If the SOS switch is detected to be manually triggered, the emergency call function is triggered.

[0058] (2) If an airbag activation signal is detected, the emergency call function is triggered.

[0059] (3) If the vehicle CAN bus data obtained through the gateway indicates that the vehicle has been in a collision, the emergency call function is triggered.

[0060] (4) If the smart cockpit recognizes the preset emergency call keywords and sends a voice keyword recognition trigger signal, the emergency call function will be triggered.

[0061] Specifically, the triggering priorities among the airbag activation signal, the CAN bus signal of a vehicle collision, the voice keyword recognition trigger signal, and the manual trigger signal, in descending order, can be: airbag activation signal, CAN bus signal of a vehicle collision (the priority of the airbag activation signal can also be the same as the priority of the CAN bus signal of a vehicle collision), voice keyword recognition trigger signal, and manual trigger signal.

[0062] One implementation method is as follows: First, obtain the highest priority trigger signal. If a high-priority trigger signal has been received, the emergency call function is triggered directly, without considering lower-priority trigger signals. If no high-priority trigger signal has been received, other trigger signals are obtained sequentially in descending order of priority (e.g., Figure 2 (as shown) Another implementation method is to determine the risk level of the emergency call function based on the number of trigger signals received. The more trigger signals received, the higher the risk level of the emergency call function, the higher the corresponding response level, and the shorter the response time.

[0063] The data terminal sends an emergency call message to the network device via a mobile communication unit. This message contains at least vehicle location information. Simultaneously or subsequently, a voice communication channel is established between the data terminal and the network device.

[0064] Optionally, it also includes: When the emergency call function is triggered, a response message is sent to the vehicle body controller, the doors are unlocked, and the hazard lights and right turn alarm are activated.

[0065] Furthermore, it also includes: The camera acquires real-time images of the vehicle's interior and sends the images to the image processing unit. The image processing unit identifies the emergency status of the occupants based on the current images (video stream information) and sends an image emergency status recognition trigger signal to the data terminal when an emergency status is detected. Emergency statuses include: driver unconsciousness or unresponsiveness, smoke or open flame inside the vehicle, and the preset triggering condition also includes the image emergency status recognition trigger signal.

[0066] The function of triggering emergency calls based on preset trigger conditions also includes: (5) If the image data collected by the camera is analyzed and identified by the image processing unit as a preset emergency state, an image emergency state identification trigger signal is sent to the data terminal, and the data terminal triggers the emergency call function.

[0067] Specifically, the trigger priorities among the airbag activation signal, the CAN bus signal of a vehicle collision, the voice keyword recognition trigger signal, the manual trigger signal, and the image emergency state recognition trigger signal, in descending order, can be as follows: airbag activation signal, CAN bus signal of a vehicle collision (the priority of the airbag activation signal can also be the same as the priority of the CAN bus signal of a vehicle collision), voice keyword recognition trigger signal, manual trigger signal, and image emergency state recognition trigger signal.

[0068] Furthermore, such as Figure 3 As shown, based on the current vehicle interior image information, identifying the emergency status of the occupants specifically includes: Acquire the vehicle's CAN bus data, which includes at least the steering wheel grip force status; Parallel analysis of emergency situations involving vehicle occupants based on computer vision algorithm models using video stream data and CAN bus data (image processing unit in the data terminal); specifically, when the emergency situation involves the driver being unconscious or unresponsive, the parallel analysis of emergency situations involving vehicle occupants based on computer vision algorithm models using video stream data and CAN bus data includes: Based on the detected facial feature points, the duration of the driver's eye closure is calculated, and the driver's unconscious or unresponsive state is identified based on the duration of the driver's eye closure and the steering wheel grip strength. When the duration of the driver's eye closure exceeds a preset time threshold (e.g., 3 seconds) and the steering wheel grip strength is characterized by the disappearance of grip strength, the driver is determined to be in an unconscious or unresponsive state.

[0069] The information such as steering wheel grip force status can be obtained through the vehicle CAN bus. This can be achieved by setting a grip force sensor (flexible thin film pressure sensor or distributed strain gauge) on the surface of the vehicle steering wheel and integrating the information collected by the grip force sensor into the original CAN bus network through a microcontroller. Alternatively, the torque signal of the electronic power steering system can be used to indirectly and continuously determine whether the driver is operating the steering wheel. This embodiment of the invention does not impose any limitations on this method. Optionally, when the emergency situation involves smoke or open flame inside the vehicle, the parallel analysis of the emergency situation of the occupants inside the vehicle based on the computer vision algorithm model includes: The system identifies whether there are visual features in the video stream that correspond to smoke or open flame. When there are visual features in the video stream that correspond to smoke or open flame, it determines that there is smoke or open flame inside the vehicle. Visual features include color features, texture features, and dynamic diffusion patterns.

[0070] Preferably, the emergency state may also include: the occupants of the vehicle experiencing pain; identification and analysis of the occupants experiencing pain: performing posture estimation on the occupants of the vehicle (applicable to situations with good lighting conditions, no obstructions, or multiple cameras at different angles), and obtaining their skeletal key points; matching the temporal data of the skeletal key points with a preset abnormal posture model (such as curling up, violently struggling), and if the matching degree exceeds a preset percentage threshold (e.g., 80%-90%, which can be customized), then it is determined that the occupant is in a state of pain.

[0071] The data terminal receives the above analysis results from the image processing unit and generates a comprehensive security status assessment, i.e., whether to trigger the emergency call function.

[0072] It should be noted that in this embodiment, the computer vision algorithm model, driver facial feature point detection model, skeletal key point acquisition model, skeletal key point matching model, and smoke or open flame state detection model can all be obtained by annotation and training based on historical image data. Moreover, the image data used for training or transmission is obtained with the user's consent and can be analyzed based on abstract facial feature points or skeletal data. There is no need to upload or store the original video images (images are only uploaded with the user's consent in emergency situations, i.e., the in-vehicle scene images or pictures captured by the camera included in the emergency call message), thus protecting the privacy of the driver inside the vehicle.

[0073] In this invention, the data terminal is communicatively connected to the SOS switch, gateway, smart cockpit, and airbag controller, respectively, and is used to trigger the emergency call function according to preset trigger conditions and send emergency call messages to network devices. The preset trigger conditions include at least one of the following: airbag action signal, CAN bus signal of vehicle collision, voice keyword recognition trigger signal, and manual trigger signal. It integrates multiple methods such as manual trigger, collision signal trigger, voice trigger, and airbag action trigger, which greatly avoids call failure due to the failure of a single sensor or the inability of personnel to operate in real emergency situations. It effectively solves the problem of low reliability and safety of vehicle emergency call triggering caused by the single emergency call triggering method in the prior art, and effectively improves the reliability and safety of vehicle emergency call triggering.

[0074] In the technical solution of this invention, the data terminal can also identify the emergency status of the occupants inside the vehicle based on the current in-vehicle image information, and send an image emergency status recognition trigger signal when the emergency status of the occupants is identified. The emergency status includes: the driver is unconscious or unresponsive, smoke or open flame is present inside the vehicle (and may also include the occupants being in pain). The preset triggering condition also includes the image emergency status recognition trigger signal, which can more accurately identify real emergency situations, effectively reduce the false alarm rate of non-emergency events, save social rescue resources, and further improve the reliability and safety of vehicle emergency call triggering.

[0075] The emergency call message in the technical solution of this invention includes: the vehicle's real-time location information, vehicle identification code, the reason for triggering the emergency call, the type of call triggered, and on-site images or pictures of the vehicle's interior captured by the camera. This enables the rescue center to understand the on-site situation of the vehicle and the condition of the people inside the vehicle in the first instance, providing key information support for formulating a precise rescue plan.

[0076] In the technical solution of this invention, when the emergency call function is triggered, a response message is simultaneously sent to the vehicle body controller to unlock the doors and activate the hazard lights and right turn alarm; this alerts passing vehicles and prevents secondary traffic accidents.

[0077] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A vehicle emergency call system based on multimodal information fusion perception, characterized in that, include: Data terminal, SOS switch, gateway, smart cockpit, and airbag controller; The airbag controller is used to acquire the airbag's activation signal; The gateway is used to acquire CAN bus signals indicating whether a vehicle collision has occurred; the smart cockpit is used to acquire currently generated voice information, including preset emergency call keywords, and to send a voice keyword recognition trigger signal; the SOS switch is used to send a manual trigger signal when an emergency call is made by occupants; the data terminal is communicatively connected to the SOS switch, gateway, smart cockpit, and airbag controller, and is used to trigger the emergency call function according to preset trigger conditions and send an emergency call message to the network device; the preset trigger conditions include at least one of the following: airbag activation signal, CAN bus signal indicating a vehicle collision, voice keyword recognition trigger signal, and manual trigger signal.

2. The vehicle emergency call system based on multimodal information fusion perception according to claim 1, characterized in that, It also includes a positioning module that is communicatively connected to the data terminal; the positioning module is used to send real-time vehicle location information to the data terminal when the emergency call function is triggered.

3. The vehicle emergency call system based on multimodal information fusion perception according to claim 2, characterized in that, It also includes a camera and an image processing unit. The camera is used to acquire real-time image information of the vehicle interior and send the image information to the image processing unit. The image processing unit is used to identify the emergency status of the occupants of the vehicle based on the current image information of the vehicle interior, and when an emergency status of the occupants of the vehicle interior is identified, it sends an image emergency status recognition trigger signal to the data terminal. The emergency status includes: the driver is unconscious or unresponsive, or there is smoke or open flame inside the vehicle. The preset triggering condition also includes the image emergency status recognition trigger signal.

4. A vehicle emergency call system based on multimodal information fusion perception according to claim 3, characterized in that, The emergency call message includes: the vehicle's real-time location information, vehicle identification number, reason for triggering the emergency call, type of call, and on-site images or pictures of the vehicle's interior captured by the camera.

5. A method for sensing vehicle emergency calls based on multimodal information fusion, characterized in that, Based on any one of claims 1-4, a multimodal information fusion perception vehicle emergency call system is implemented, comprising: The airbag controller receives the activation signal of the airbag; The gateway obtains the CAN bus signal indicating whether a vehicle collision has occurred; The intelligent cockpit acquires currently generated voice information, which includes preset emergency call keywords, and sends a voice keyword recognition trigger signal. The SOS switch sends a manual trigger signal when an emergency call is made by someone inside the vehicle. The data terminal triggers the emergency call function according to preset trigger conditions and sends an emergency call message to the network device; the preset trigger conditions include at least one of the following: airbag activation signal, CAN bus signal of vehicle collision, voice keyword recognition trigger signal, and manual trigger signal.

6. The method for vehicle emergency call based on multimodal information fusion according to claim 5, characterized in that, Also includes: When the emergency call function is triggered, a response message is sent to the vehicle body controller, the doors are unlocked, and the hazard lights and right turn alarm are activated.

7. The method for vehicle emergency call based on multimodal information fusion according to claim 5, characterized in that, Also includes: The camera acquires real-time images of the vehicle's interior and sends these images to the image processing unit. The image processing unit identifies the emergency status of the occupants inside the vehicle based on the current image information inside the vehicle, and sends an image emergency status recognition trigger signal to the data terminal when an emergency status is identified. The emergency status includes: the driver is unconscious or unresponsive, or there is smoke or open flame inside the vehicle. The preset triggering condition also includes the image emergency status recognition trigger signal.

8. The method for sensing vehicle emergency calls based on multimodal information fusion according to claim 7, characterized in that, Based on the current image information inside the vehicle, identifying the emergency status of the occupants specifically includes: Acquire the vehicle's CAN bus data, which includes at least the steering wheel grip force status; Parallel analysis of emergency states for vehicle occupants based on computer vision algorithm models using video stream data and CAN bus data; specifically, when the emergency state is that the driver is unconscious or unresponsive, the parallel analysis of emergency states for vehicle occupants based on computer vision algorithm models using video stream data and CAN bus data includes: Based on the detected facial feature points, the duration of the driver's eye closure is calculated, and the driver's unconscious or unresponsive state is identified based on the duration of the driver's eye closure and the steering wheel grip strength. When the duration of the driver's eye closure exceeds a preset threshold and the steering wheel grip strength is characterized by the disappearance of grip strength, the driver is determined to be in an unconscious or unresponsive state.

9. A method for sensing vehicle emergency calls based on multimodal information fusion according to claim 8, characterized in that, When an emergency situation occurs, such as smoke or open flame inside the vehicle, the parallel analysis of the emergency situation of the occupants inside the vehicle based on computer vision algorithm models and video stream data and CAN bus data specifically includes: The system identifies whether there are visual features in the video stream that correspond to smoke or open flame. When there are visual features in the video stream that correspond to smoke or open flame, it determines that there is smoke or open flame inside the vehicle. Visual features include color features and texture features.

10. A method for sensing vehicle emergency calls based on multimodal information fusion according to claim 8, characterized in that, The emergency state also includes: the occupants of the vehicle are in a state of distress; when the emergency state is that the occupants of the vehicle are in a state of distress, the parallel analysis of the emergency state of the occupants of the vehicle based on the computer vision algorithm model on the video stream data and CAN bus data specifically includes: By estimating the posture, the skeletal key point data of the occupants in the vehicle is obtained, and the skeletal key point data of the occupants in the vehicle is matched with a preset abnormal posture model. If the matching degree exceeds a preset percentage threshold, the occupants in the vehicle are determined to be in a state of pain.

Images