Road alarm help positioning early warning system based on multi-modal interaction

Abstract

The application discloses a road alarm help positioning early warning system based on multi-modal interaction and relates to the technical field of intelligent traffic and safety emergency.The system comprises an alarm help positioning early warning platform and a mobile phone terminal information acquisition program.Through the integration of three modal data of voice, image and map positioning, a feature extraction and matching model is constructed to form a three-in-one information closed loop of voice description event nature, image intuitive presentation of the scene and positioning accurate locking of the position.The voice interaction can quickly capture key semantics, make up for the deficiency that image cannot directly transmit text information, image recognition can intuitively display the scene details, solve the information loss caused by noise interference or user tension in voice description, and map positioning provides position coordinates through GPS technology to perform feature matching and reliability calculation on the three modal data, realize cross verification of multi-source data, significantly reduce disposal delay caused by information deviation, and provide more reliable decision basis for rescue departments.

Description

Technical Field

[0001] This invention relates to the field of intelligent transportation and emergency safety technology, specifically a road alarm, assistance, location, and early warning system based on multimodal interaction. Background Technology

[0002] With the continuous advancement of road construction and the rapid increase in vehicle ownership, emergencies such as road damage, traffic accidents, and deterioration of road conditions caused by severe weather occur frequently during road use. Timely and effective emergency response is crucial in these situations. However, in real-world scenarios, individuals often struggle to accurately provide their location due to panic, unfamiliarity with the environment, or poor ability to describe their geographical location. This hinders rapid response and precise command in road rescue efforts. Furthermore, delays, misunderstandings, or incomplete information transmission can further delay the response, reduce efficiency, and may even lead to more serious consequences.

[0003] In the field of intelligent transportation and emergency response, although voice and image interaction technologies have made some progress, many problems still exist. Regarding voice interaction, firstly, information is incomplete and easily affected by environmental noise; when users are nervous or lack sufficient language skills, information is often incomplete, such as only stating "an accident has occurred ahead" without specifying the lane or vehicle type. Secondly, positioning accuracy is low; relying solely on voice-described location information makes accurate positioning difficult in environments without clear landmarks or complex road conditions, resulting in significant errors. Regarding image interaction, firstly, it has poor adaptability to dynamic scenes; existing image processing technologies have low accuracy in recognizing blurry, backlit, or moving targets, relying on high-resolution equipment and stable shooting conditions. Secondly, semantic information is lacking; images cannot directly convey the contextual information described in the voice, leading to delays in emergency response.

[0004] Existing technologies suffer from poor environmental adaptability and low information fusion in multimodal interaction. Therefore, there is a need for a road emergency call positioning and early warning system based on multimodal interaction. This system can overcome the shortcomings of existing interaction methods and improve the efficiency and accuracy of road emergency calls by integrating voice, image and map positioning. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a road emergency call location and early warning system based on multimodal interaction. It can integrate voice, image, and map positioning data to form complementary advantages and solve the core pain points of traditional single-modal systems, such as incomplete information, large positioning deviation, and high response delay. When the handling center receives an alarm call from the party involved in a road incident, the system will trigger a response mechanism to ensure the accuracy and timeliness of information transmission, thereby making a more reasonable response decision.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: a road alarm assistance location and early warning system based on multimodal interaction. The system includes an alarm assistance location and early warning platform and a mobile information collection program. It integrates voice description, on-site images and map positioning data through multimodal fusion technology, establishes a multimodal feature matching model, and generates comprehensive alarm information through data association.

[0007] The alarm assistance location and early warning platform includes a system monitoring module, an alarm assistance location module, an early warning SMS notification module, a query and statistics module, and a system management module.

[0008] The mobile information collection program consists of multiple functional interfaces. The person reporting the incident receives a text message containing a unique link. After clicking the link, the system automatically obtains GPS location information, supports uploading on-site photos and videos, and supplements the accident description. After submission, the information is synchronized to the alarm assistance location and early warning platform in real time, completing the collection of alarm information.

[0009] Furthermore, the system monitoring module consists of online user and service monitoring, which displays the online user status and platform resource usage in real time;

[0010] The online user information is displayed in real time by the system.

[0011] The service monitoring system displays the system resources used by the platform in real time.

[0012] Furthermore, the alarm assistance and location module sends a unique link to the caller's mobile phone via SMS based on the caller's mobile phone number, triggering GPS positioning and image upload functions.

[0013] Furthermore, the warning SMS notification module is used to send and query SMS reminders for fatigued driving and illegal parking, and supports customized content push.

[0014] Furthermore, the query and statistics module consists of location feedback query, location feedback statistics, warning SMS query, and warning SMS statistics, which are used to provide visualized statistics on alarm records, location feedback, and SMS sending volume;

[0015] The location feedback query is used to view alarm information, delete information, upload images, and export to Excel.

[0016] The location feedback statistics are displayed in a bar chart showing the number of SMS messages sent by alarm personnel in each department to provide location feedback.

[0017] The warning SMS query is used to check the sending status of warning SMS messages;

[0018] The aforementioned warning SMS statistics are used to query statistical bar charts of warning SMS messages.

[0019] Furthermore, the system management module includes user management, role management, menu management, department management, job management, dictionary management, parameter settings, notifications and announcements, and log management, and supports multi-level department data permission configuration;

[0020] The user management system is used to maintain basic user information and allocate data and function permissions.

[0021] The role management is used to maintain basic role information and assign functional permissions;

[0022] The menu management system is used to manage system menu buttons, and users with different permissions will display different menu function items.

[0023] The department management system allows users to add their assigned departments, set up a multi-level tree structure, and assign data permissions.

[0024] The job management feature allows users to add jobs and assign data permissions.

[0025] The dictionary management manages the data dictionary used in the system and supports dictionary mapping for other data tables in the system;

[0026] The parameter settings are used to manage the default parameter settings in the system, including the initial password and menu style;

[0027] The aforementioned notices and announcements are used for editing and managing daily notifications and news items on the platform;

[0028] The log management is divided into operation logs and login logs, which can be used to view the functional modules that operators have entered and their login status.

[0029] Furthermore, the process by which the multimodal fusion technology integrates voice description, on-site images, and map positioning data is as follows:

[0030] It receives voice description data input by the user, real-time captured on-site image data, and map positioning data provided by the mobile phone.

[0031] Semantic parsing is performed on the voice description data to extract semantic features including event type, landmark references, and hazard level;

[0032] Dynamic target detection and scene segmentation are performed on the on-site image data to extract image features of vehicle position, road signs and environmental conditions;

[0033] The map positioning data is matched with the road network database to generate three-dimensional location information in a spatial coordinate system;

[0034] The semantic features, image features, and three-dimensional location information are matched and associated using a multimodal feature matching model to generate comprehensive alarm information.

[0035] Based on the event type and location coordinates in the comprehensive alarm information, the system automatically associates the preset emergency response strategy and pushes it to the target terminal.

[0036] Furthermore, the multimodal feature matching model is used to match and associate the semantic features, image features, and three-dimensional location information, specifically including:

[0037] Set matching rules, and based on the accident type keywords in the semantic features, find the corresponding accident vehicles and damage in the image features, and match the specific road segment where the accident occurred in the three-dimensional location information;

[0038] Using an association algorithm, successfully matched voice features, image features, and 3D location information are associated to form a data set with logical relationships. Based on this data set, comprehensive alarm information is generated. The association algorithm is built on a Bayesian network and is used to evaluate the association confidence between each feature. Specifically, the association confidence between semantic features (V), image features (I), 3D location information (L), and target event C is set as P(C|V,I,L). Where P(C) is the prior probability of the target event C, i.e., the probability of event C occurring in historical data; P(V,I,L|C) is the joint likelihood of observed features V,I,L given the target event C; and P(V,I,L) is the joint prior probability of features V,I,L, and P(V,I,L)=∑ C′ P(V,I,L|C′)·P(C′), where C′ represents all event types, and P(C|V,I,L) is the posterior probability, representing the probability of target event C occurring when features V,I,L are observed. This is used to quantify the credibility of the association between features and events. When P(C|V,I,L)≥θ, where θ is a preset credibility threshold, the successfully matched V,I,L features are associated as a set of valid data to generate comprehensive alarm information containing accident location, type, and severity information.

[0039] Compared with existing technologies, this road emergency call location and warning system based on multimodal interaction has the following advantages:

[0040] I. This invention integrates three modalities of data—voice, image, and map positioning—to construct a feature extraction and matching model, forming a three-in-one information loop: voice describing the nature of an event, images intuitively presenting the scene, and precise positioning pinpointing the location. Voice interaction can quickly capture key semantics, compensating for the inability of images to directly convey textual information. Image recognition can intuitively display scene details, solving the problem of information loss caused by noise interference or user anxiety in voice descriptions. Map positioning provides location coordinates through GPS technology. By performing feature matching and credibility calculation on the three modalities, cross-validation of multi-source data is achieved, significantly reducing delays in response due to information bias and providing rescue departments with more reliable decision-making basis.

[0041] Second, this system has built a highly efficient interactive system through an automated process of telephone triggering, customized SMS guidance, and multimodal information collection. When the emergency response center receives an alarm, it automatically sends an SMS containing a unique link to the user. The user can click the link to obtain the location in real time. At the same time, it supports selective uploading of on-site photos or videos, which greatly reduces the difficulty of operation for users in emergency situations and improves the convenience of users reporting alarms. Furthermore, it enables rapid collaboration through standardized information formats, effectively improving the overall efficiency of road emergency response.

[0042] Other advantages, objectives and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination or study, or may be learned from the practice of the invention. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0044] Figure 1 This is an operation flowchart for a road emergency call location and early warning system based on multimodal interaction;

[0045] Figure 2 This is a system composition diagram of a road emergency call location and early warning system based on multimodal interaction;

[0046] Figure 3 This is a flowchart of the alarm information collection process on the mobile terminal of a road alarm assistance location and early warning system based on multimodal interaction. Detailed Implementation

[0047] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.

[0048] Example 1

[0049] This embodiment focuses on the operation mechanism of a road alarm, assistance, location, and early warning system based on multimodal interaction. Through system breakdown and detailed process description, it deeply analyzes the working principle of each module and presents how the system integrates voice, image, and map positioning data to achieve efficient alarm, assistance, location, and early warning, providing precise support for road safety emergency response.

[0050] When the system initiates the alarm and assistance process, the alarm and assistance location module of the alarm and assistance location and early warning platform plays a crucial role. This module generates a unique SMS link based on the received alarm and assistance caller's mobile phone number. This link accurately identifies each alarm event, avoiding information confusion. Figure 3 As shown, after the SMS message is sent, once the caller's mobile phone receives an SMS containing a unique link, clicking the link will trigger the mobile information collection program. Once activated, the program uses the phone's GPS map location data. The program allows the caller to upload photos and videos of the scene and add a description of the accident. The caller can use their phone's camera to capture images of the accident scene, recording key information such as vehicle condition, road conditions, and surrounding signs. They can also input text descriptions on the phone's interface, such as the type of accident and its general circumstances. These image, video, and text data, along with the previously acquired GPS location data, constitute multimodal information. After information collection is complete, clicking the submit button will cause the mobile information collection program to transmit the integrated multimodal information in real time to the alarm and assistance location warning platform via network communication protocols, completing the initial collection and uploading of the alarm information.

[0051] like Figure 2As shown, the alarm assistance and location early warning platform includes a system monitoring module, an alarm assistance location module, an early warning SMS notification module, a query and statistics module, and a system management module. The system monitoring module consists of two collaborative functions: online user monitoring and service monitoring. The online user monitoring function is responsible for real-time monitoring of system access. Whenever a new user accesses the system via mobile phone, their relevant information, such as mobile phone number, access time, and device identifier, is accurately recorded and displayed on the monitoring interface. This allows platform administrators to grasp the current system usage at any time and promptly identify anomalies or potential problems. The service monitoring function focuses on the platform's own operational status. It continuously monitors various resource indicators during platform operation, including but not limited to CPU utilization, memory usage, and network bandwidth consumption. Through real-time analysis of these indicators, the service monitoring function can intelligently adjust platform resource allocation to ensure stable and efficient operation even when alarm information floods in or system load increases, avoiding problems such as lag and crashes. The query and statistics module uses a location feedback query function for detailed retrieval and management of alarm information. It accurately extracts alarm information from the platform database. The platform displays various information uploaded by police officers, including location data, images, and text descriptions, in an intuitive and clear interface. Staff can perform multi-dimensional operations on this interface, such as viewing complete alarm information, deleting erroneous or redundant information, supplementing missing images, and exporting relevant information to Excel files for further analysis and data organization. The location feedback statistics function collects and analyzes location feedback data from alarm personnel in various departments, displaying the number of SMS messages sent by each department in the form of statistical bar charts. This visualization helps platform administrators intuitively understand the work efficiency and response speed of different departments in the alarm handling process, thus providing data support for optimizing departmental collaboration and improving overall emergency response capabilities. The warning SMS query function allows staff to check the sending status of warning SMS messages. Staff can obtain detailed information such as the sending time, recipients, and sending results of each warning SMS message, facilitating the tracking of the dissemination and reception of warning information. The warning SMS statistics function presents the changing trends in the number of different types of warning SMS messages sent in the form of statistical bar charts, helping the platform evaluate the effectiveness of warning information dissemination and providing a basis for subsequent optimization of warning strategies.

[0052] The aforementioned warning SMS notification module intelligently determines whether to send a warning SMS and determines the SMS content based on the platform's preset rules and received alarm information. For example, in this embodiment, when the platform receives an alarm about an accident affecting traffic flow, the warning SMS notification module will automatically generate a warning SMS containing information such as the accident location, the expected impact range, and traffic control suggestions, and send it to drivers of surrounding vehicles. This module not only supports sending and querying conventional warning SMS such as those for fatigued driving and illegal parking, but also has a custom content push function, which can flexibly adjust the warning information according to the actual situation to meet diverse warning needs.

[0053] The system management module includes user management, role management, menu management, department management, job management, dictionary management, parameter settings, notifications and announcements, and log management. It supports multiple functions such as multi-level departmental data permission configuration. The user management function maintains basic user information, including name, contact information, account password, etc., and finely allocates data access and function operation permissions according to user responsibilities and permission requirements. For example, ordinary alarm handlers can only view and process alarm information, while system administrators have higher permissions and can perform operations such as system parameter settings and user account management. The role management function defines and maintains different roles in the system and assigns corresponding function permissions to each role, ensuring that users can only perform operations they are authorized to perform, enhancing system security and stability. The menu management function dynamically manages system menu buttons based on user permission settings. Users with different permissions will see different menu function items after logging in, further refining user operation permissions and improving the convenience and security of system operation. The department management function adds the user's affiliated department and constructs a multi-level tree structure. This approach assigns data permissions to users in different departments, enabling data isolation and collaborative work between departments. The job management function further refines data permission allocation based on users' job requirements, ensuring that each user can only access and process data related to their work. The dictionary management function provides unified management of the data dictionary used in the system, offering standardized dictionary mappings for other data tables to ensure data consistency and standardization. The parameter setting function manages default parameters for the system, such as initial password rules, menu display styles, and data storage paths, which can be flexibly adjusted according to actual usage scenarios and user needs. The notification and announcement function is responsible for editing and publishing daily notifications and news on the platform, such as system upgrade announcements and emergency event notifications, ensuring that users can obtain important information in a timely manner. The log management function is divided into operation logs and login logs. The operation log records in detail each user's operation steps, operation time, and operation objects in the system, facilitating operation traceability and problem troubleshooting. The login log records the user's login time, IP address, login device, and other information to monitor system access and ensure system security.

[0054] After receiving voice description data, on-site image data, and map positioning data uploaded from a mobile device, the alarm and emergency location warning platform uses multimodal fusion technology to perform semantic analysis on the voice description data, identifying key semantic features such as event type, landmark references, and hazard level. For the on-site image data, the platform performs dynamic target detection and scene segmentation, quickly identifying moving objects such as accident vehicles and pedestrians, determining their position, speed, and trajectory, and dividing the image into different regions such as roads, buildings, and green belts. It extracts image features such as road markings (lane lines, traffic signs) and environmental conditions (smoke, water accumulation), obtaining rich on-site information from the images to provide strong support for subsequent analysis. Upon receiving the map positioning data, the platform matches it with a pre-built road network database containing detailed road information such as road name, direction, number of lanes, and surrounding landmarks. This is achieved by converting the GPS coordinates obtained from the mobile device into specific geographic information. Location information is generated in a spatial coordinate system to form three-dimensional location information, including the specific road segment where the accident occurred, its distance and orientation from surrounding landmarks, etc., making the accident location more accurate and intuitive. To achieve effective fusion of multimodal data, the platform adopts a multimodal feature matching model. This model sets a series of matching rules. Based on the accident type keywords in the semantic features, it searches for the corresponding accident vehicle and damage in the image features, and matches the specific road segment where the accident occurred in the three-dimensional location information. For example, in this embodiment, if the semantic features mention "rear-end collision," the model will search for images of vehicle rear-end collisions in the image features and locate the exact road segment where the accident occurred in the three-dimensional location information. The model uses an association algorithm based on a Bayesian network to associate the successfully matched speech features, image features, and three-dimensional location information. The association confidence of semantic features (V), image features (I), and three-dimensional location information (L) with the target event C (such as the specific details and severity of the accident) is set as P(C|V,I,L), calculated by the formula: Wherein, P(C) is the prior probability of the target event C, which is the probability of event C occurring based on historical data statistics. For example, for the probability of a "rear-end collision" occurring on this road segment, the system performs statistical calculations by analyzing historical alarm data and accident records. P(V,I,L|C) is the joint likelihood of observed features V, I, and L given the target event C, reflecting the probability of the current voice description, image, and location information appearing when it is known that the accident is of this type. In this embodiment, P(V,I,L|C) is decomposed into P(V,I,L|C) = P(V|C)·P(I|C)·P(L|C). For the voice feature conditional probability P(V|C), the semantic keyword feature vector V = {v1,v2,…,v} is extracted. mIn the historical cases of event C, the keyword v i The frequency of occurrence, if V contains m keywords, and each keyword is independent, then P Where P(v) i |C) represents the keyword v in event C. i Similarly, the probability of occurrence is the image feature conditional probability P(I|C) obtained by extracting the image feature vector I = {i1, i2, ..., i...}. n}, Image features i in statistical event C j Frequency of occurrence The location feature conditional probability P(L|C) is calculated using the location feature vector L = {l1, l2, l3}, including latitude and longitude l1, road type l2, and landmark distance l3, to obtain P(L|C) = P(l1|C)·P(l2|C)·P(l3|C). This allows for the calculation of the joint likelihood P(V,I,L|C) for features V, I, and L, where P(V,I,L) is the joint prior probability of features V, I, and L, expressed by the formula P(V,I,L) = ∑ C′ The formula P(V,I,L|C′)·P(C′) is used to calculate the probability of occurrence of the current feature. C′ represents all possible event types. This formula comprehensively considers the impact of all possible events on the probability of occurrence of the current feature. P(C|V,I,L) is used as the posterior probability to quantify the credibility of the association between the feature and the event. When P(C|V,I,L)≥θ, θ is a preset credibility threshold. The system associates the successfully matched V, I, and L features into a set of valid data and generates comprehensive alarm information containing information on the location, type, and severity of the accident.

[0055] Finally, based on the event type and location coordinates in the generated comprehensive alarm information, the platform automatically associates the preset emergency response strategy and pushes the relevant information to the target terminal. The platform has pre-stored a variety of emergency response strategies for different types of accidents and locations. The platform pushes alarm commands containing detailed accident information to the terminal devices of the relevant departments, such as handheld terminals and vehicle computers of rescue vehicles, to ensure that the relevant departments can respond quickly and carry out emergency handling work.

[0056] In summary, this embodiment details the workflow of a road emergency call location and warning system based on multimodal interaction. It covers the entire process from mobile phone information collection to platform data processing, multimodal fusion analysis, and emergency response push. The various modules of the system work closely together, achieving efficient processing and accurate location warning of road emergency call information through multimodal fusion technology. The mobile phone information collection program conveniently acquires multimodal data, and the various modules of the platform manage, analyze, and fuse the data in all aspects. The multimodal feature matching model uses Bayesian algorithms to ensure the accuracy of data association, ultimately achieving a fast and accurate emergency response.

[0057] Example 2

[0058] This embodiment applies a multimodal interactive road emergency call location and early warning system to traffic accidents. The specific steps for the parties involved to call for help using this system are as follows:

[0059] Triggering the alarm: The person involved dials the emergency number or initiates the alarm process through a mobile app, sending a distress signal to the response center.

[0060] Receive SMS link: The processing center sends a customized SMS containing a unique link to the person's mobile phone through the system platform.

[0061] Click the link to start the data collection process: When the person clicks the link in the text message, they will be automatically redirected to the information collection interface on their mobile phone.

[0062] Automatically obtain accurate location: The system automatically obtains the latitude and longitude coordinates of the accident location through the mobile phone's GPS and displays them on the map for user confirmation.

[0063] Take and upload on-site images / videos: The parties involved use their mobile phone cameras to take photos or videos of the accident scene (multiple consecutive shots are supported), and click the "Upload" button to submit them to the platform.

[0064] Additional accident description (optional): The parties involved can record a voice description of the accident type and fill in a brief text description.

[0065] Confirm and submit information: After the party concerned checks that the location, image and supplementary information are correct, click the "Submit" button to complete the alarm information collection.

[0066] The platform automatically processes and distributes information: The system integrates location, image, and description information into a structured alarm report, automatically pushes it to relevant departments, and generates a response plan.

[0067] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A road emergency call location and early warning system based on multimodal interaction, characterized in that, The system includes an alarm assistance location and early warning platform and a mobile information collection program. It integrates voice description, on-site images and map location data through multimodal fusion technology, establishes a multimodal feature matching model, and generates comprehensive alarm information through data association. The alarm assistance location and early warning platform includes a system monitoring module, an alarm assistance location module, an early warning SMS notification module, a query and statistics module, and a system management module. The mobile information collection program consists of multiple functional interfaces. The alarm person receives a text message containing a unique link. After clicking the link, the GPS location information is automatically obtained. It supports uploading on-site pictures and videos, supplementing the accident description, and after submission, the information is synchronized to the alarm assistance location and early warning platform in real time to complete the collection of alarm information. The multimodal feature matching model is used to match and associate semantic features, image features, and 3D location information, specifically including: Set matching rules, and based on the accident type keywords in the semantic features, find the corresponding accident vehicles and damage in the image features, and match the specific road segment where the accident occurred in the three-dimensional location information; Using an association algorithm, successfully matched semantic features, image features, and 3D location information are associated to form a data set with logical relationships. Based on this data set, comprehensive alarm information is generated. The association algorithm is built on a Bayesian network and is used to evaluate the reliability of the association between each feature; that is, the semantic features are set as... Image features are Three-dimensional location information is , and the target event The reliability of the association is and ,in, For the target event The prior probability, that is, the event in historical data. The probability of occurrence For a given target event At that time, the observed features The joint likelihood Features The joint prior probabilities and , For all event types, Let be the posterior probability, representing the probability after observing the feature. At that time, the target event The probability of occurrence is used to quantify the reliability of the association between features and events. hour, To preset a confidence threshold, successfully matched items will be... Feature association is used to generate a set of valid data, which in turn generates comprehensive alarm information including the location, type, and severity of the accident.

2. The road emergency call location and early warning system based on multimodal interaction according to claim 1, characterized in that, The system monitoring module consists of online user and service monitoring, which displays the online user status and platform resource usage in real time. The online user information is displayed in real time by the system. The service monitoring system displays the system resources used by the platform in real time.

3. The road emergency call location and early warning system based on multimodal interaction according to claim 1, characterized in that, The alarm assistance and location module sends a unique link to the caller's mobile phone via SMS based on the caller's mobile phone number, triggering GPS positioning and image upload functions.

4. The road emergency call location and early warning system based on multimodal interaction according to claim 1, characterized in that, The warning SMS notification module is used to send and query SMS reminders for fatigued driving and illegal parking, and supports customized content push.

5. The road emergency call location and early warning system based on multimodal interaction according to claim 1, characterized in that, The query and statistics module consists of location feedback query, location feedback statistics, warning SMS query, and warning SMS statistics, and is used to provide visual statistics on alarm records, location feedback, and SMS sending volume. The location feedback query is used to view alarm information, delete information, upload images, and export to Excel. The location feedback statistics are displayed in a bar chart showing the number of SMS messages sent by alarm personnel in each department to provide location feedback. The warning SMS query is used to check the sending status of warning SMS messages; The aforementioned warning SMS statistics are used to query statistical bar charts of warning SMS messages.

6. The road emergency call location and early warning system based on multimodal interaction according to claim 1, characterized in that, The system management module includes user management, role management, menu management, department management, position management, dictionary management, parameter settings, notifications and announcements, and log management, and supports multi-level department data permission configuration; The user management system is used to maintain basic user information and allocate data and function permissions. The role management is used to maintain basic role information and assign functional permissions; The menu management system is used to manage system menu buttons, and users with different permissions will display different menu function items. The department management system allows users to add their assigned departments, set up a multi-level tree structure, and assign data permissions. The job management feature allows users to add jobs and assign data permissions. The dictionary management manages the data dictionary used in the system and supports dictionary mapping for other data tables in the system; The parameter settings are used to manage the default parameter settings in the system, including the initial password and menu style; The aforementioned notices and announcements are used for editing and managing daily notifications and news items on the platform; The log management is divided into operation logs and login logs, which can be used to view the functional modules that operators have entered and their login status.

7. The road emergency call location and early warning system based on multimodal interaction according to claim 1, characterized in that, The process by which the multimodal fusion technology integrates voice description, on-site images, and map positioning data is as follows: It receives voice description data input by the user, real-time captured on-site image data, and map positioning data provided by the mobile phone. Semantic parsing is performed on the voice description data to extract semantic features including event type, landmark references, and hazard level; Dynamic target detection and scene segmentation are performed on the on-site image data to extract image features of vehicle position, road signs and environmental conditions; The map positioning data is matched with the road network database to generate three-dimensional location information in a spatial coordinate system; The semantic features, image features, and three-dimensional location information are matched and associated using a multimodal feature matching model to generate comprehensive alarm information. Based on the event type and location coordinates in the comprehensive alarm information, the system automatically associates the preset emergency response strategy and pushes it to the target terminal.

Images