Method and system for reminding violation of traffic laws and regulations based on multi-mode large model

Abstract

The invention provides a traffic law violation reminding method and system based on a multi-modal large model, and belongs to the technical field of automobile intelligent driving, the method comprises the following steps: obtaining multi-modal data of a current traffic scene, the multi-modal data of the current traffic scene comprising image data, audio data and text data; inputting the multi-modal data of the current traffic scene into a pre-trained multi-modal large model, and performing real-time reasoning and outputting a reasoning result by the pre-trained multi-modal large model; and generating traffic regulation violation reminding information according to the reasoning result. The comprehensive road scene understanding ability is constructed by fusing multi-source data such as vehicle-mounted vision, hearing, positioning and vehicle state. Compared with a traditional single-mode identification scheme, the traffic violation detection accuracy can be greatly improved in complex driving environments (such as severe weather and dense building groups), and the problems of view limitation and insufficient environmental adaptability of a single sensor are effectively solved.

Description

Technical Field

[0001] This invention relates to the field of intelligent driving technology for automobiles, and in particular to a method and system for alerting drivers of traffic violations based on a multimodal large model. Background Technology

[0002] In modern society, with the continuous advancement of technology, the sustained growth of vehicle ownership, and the increasing complexity of traffic scenarios, accurate and real-time monitoring and alerts for traffic violations have become increasingly important. This is especially true for novice drivers. With more vehicles on the road, the amount of information required increases. Attention can be drawn to pedestrians or other vehicles, causing them to overlook traffic signs such as traffic lights and double yellow lines. This overload of information can lead to a loss of focus and a risk of violating traffic regulations.

[0003] The enforcement of existing traffic regulations relies primarily on traditional technologies, which have revealed numerous limitations when dealing with complex and ever-changing traffic environments. Furthermore, in overly simplistic traffic scenarios, while drivers may be perfectly capable of driving normally, they may still commit traffic violations such as running red lights or not driving in designated lanes due to familiarity with the route and road conditions, or distraction from pets, mobile phone messages, or other distractions. This not only causes losses for the driver but also poses safety risks to other vehicles and pedestrians on the road.

[0004] Regarding voice prompts, existing in-vehicle voice systems, while providing some prompts when intelligent driving or navigation is activated, are mostly based on preset rules and fixed scenarios, lacking the ability to dynamically perceive and flexibly respond to real-time traffic conditions. For example, common in-vehicle navigation voice prompts can only provide alerts based on pre-recorded information such as fixed speed camera locations and restricted areas, and cannot respond in real time to temporary traffic control, sudden road conditions, or immediate violations by drivers. When encountering lane changes due to temporary construction or special traffic rules directed by traffic police on-site, such systems often fail to provide effective reminders.

[0005] Furthermore, existing traffic violation monitoring and voice reminder systems are mostly based on single-modal data processing technology, lacking the ability to comprehensively analyze and understand multi-source information. For example, although traffic signs such as speed limits and no-parking signs may be recognized and displayed on the instrument panel, drivers may not necessarily notice them. Taking image recognition as an example, relying solely on camera images makes it difficult to simultaneously and comprehensively perceive the vehicle's driving status, driver behavior, the dynamics of surrounding traffic participants, and traffic signs and environmental information. In complex scenarios, a single information source can easily lead to missing information or misjudgments, failing to accurately capture potential violations. Moreover, when processing traffic regulations, these systems often rely on simple rule matching, lacking the ability to understand and reason about the deeper semantics of legal provisions, making it difficult to cope with complex and ever-changing real-world traffic scenarios.

[0006] Therefore, how to achieve real-time recognition and alerts for dynamic traffic scenarios such as temporary signs, and fill the gap in reliance on fixed data, has become a key technical problem that urgently needs to be solved. Summary of the Invention

[0007] This invention aims to solve at least one of the aforementioned problems in the prior art. By using multimodal fusion to enhance anti-interference capabilities and improve the recognition accuracy of traffic scenarios in complex environments, it achieves a leap from "feature recognition" to "semantic understanding." It can perform logical reasoning based on traffic law knowledge graphs and provide explanatory reminders. It also optimizes the driver interaction experience, supports natural language conversational reminders, and meets personalized rule query needs.

[0008] In a first aspect, embodiments of the present invention provide a method for alerting traffic violations based on a multimodal large model, including:

[0009] Acquire multimodal data of the current traffic scene, which includes image data, audio data, and text data;

[0010] The multimodal data of the current traffic scenario is input into a pre-trained multimodal large model, which then performs real-time inference and outputs the inference results.

[0011] Traffic violation alerts are generated based on the reasoning results.

[0012] In a preferred embodiment, the method further includes: a step of training a multimodal large model; the step of training the multimodal large model includes:

[0013] Acquire multimodal sample data from multiple different traffic scenarios;

[0014] The multimodal sample data from the various traffic scenarios are input into a pre-defined multimodal large model for training.

[0015] Real-world violation cases are introduced into a pre-set multimodal large model for fine-tuning to obtain the pre-trained multimodal large model;

[0016] The pre-defined multimodal large model architecture adopts an encoder-decoder structure, wherein the encoder includes: a visual Transformer for processing image data, a speech Transformer for processing audio data, and a text Transformer for processing text data; the pre-defined multimodal large model achieves feature fusion through a cross-modal attention mechanism.

[0017] In a preferred embodiment, the image data includes: road images captured by an in-vehicle camera; the audio data includes: in-vehicle sound data and / or external ambient sound data; and the text data includes: a traffic regulation knowledge graph, location and status data.

[0018] In a preferred embodiment, before the step of inputting the multimodal data of the current traffic scenario into a pre-trained multimodal large model, and the pre-trained multimodal large model performing real-time inference and outputting inference results, the method includes: a multimodal data preprocessing step for the current traffic scenario; the multimodal data preprocessing step for the current traffic scenario includes:

[0019] The image data is denoised and distortion corrected.

[0020] The audio data is subjected to noise reduction and sound source localization.

[0021] The location and status data are matched with a high-precision map to obtain basic road segment information.

[0022] In a preferred embodiment, the step of inputting the multimodal data of the current traffic scenario into a pre-trained multimodal large model, and the pre-trained multimodal large model performing real-time inference and outputting inference results includes:

[0023] The pre-trained multimodal large model identifies at least one of traffic signs, traffic light status, road markings, and temporary construction signs through image data and outputs visual semantic labels;

[0024] The pre-trained multimodal large model identifies speech features through audio data;

[0025] The pre-trained multimodal large model performs fusion reasoning based on visual semantic labels, speech features, localization and state data, and traffic regulations knowledge graph. The output reasoning results include: violation type and the mapping relationship between violation type and regulation clause.

[0026] In a preferred embodiment, generating the traffic violation reminder information based on the reasoning result includes:

[0027] Personalized reminders are generated based on the type and severity of the violation and the driving scenario.

[0028] The personalized reminder will be sent to the car's speakers for playback.

[0029] In a preferred embodiment, the method further includes: a voice query step; the voice query step includes:

[0030] When a voice query request is received, the system outputs the explanation of the corresponding legal clause based on the mapping relationship between violation type and legal clause.

[0031] Secondly, embodiments of the present invention provide a traffic violation alert system based on a multimodal large model, configured to implement any of the methods described in the first aspect, the system comprising:

[0032] The acquisition module is used to acquire multimodal data of the current traffic scene, which includes image data, audio data, and text data.

[0033] The model processing module is used to input the multimodal data of the current traffic scenario into a pre-trained multimodal large model, and the pre-trained multimodal large model performs inference in real time and outputs the inference results.

[0034] The reminder module is used to generate traffic violation reminder information based on the reasoning results.

[0035] Thirdly, embodiments of the present invention provide an electronic device, including:

[0036] One or more processors;

[0037] Memory, used to store one or more programs;

[0038] When the one or more programs are executed by the one or more processors, the one or more processors implement the methods as described in any of the first aspects.

[0039] Fourthly, embodiments of the present invention provide a computer-readable medium storing a computer program that, when executed by a processor, implements the steps of any of the methods described in the first aspect.

[0040] Beneficial effects of this invention:

[0041] 1. Comprehensively expand the dimensions of in-vehicle environmental perception

[0042] This invention integrates multi-source data, including in-vehicle vision, hearing, positioning, and vehicle status, to build a comprehensive road scene understanding capability. Compared to traditional single-modal recognition solutions, it can significantly improve the accuracy of traffic violation detection in complex driving environments (such as inclement weather or dense building complexes), effectively compensating for the limitations of a single sensor's field of view and insufficient environmental adaptability.

[0043] 2. Achieve real-time response to dynamic traffic rules.

[0044] Leveraging the real-time inference capabilities of a multimodal large model, this invention can simultaneously process vehicle sensor data and real-time updated traffic regulations, enabling rapid responses to dynamic scenarios such as temporary construction restrictions and driver inattention on familiar routes. Compared to traditional solutions that rely on pre-stored data, it effectively covers more dynamic violation scenarios that were previously difficult to handle, avoiding alert failures due to data update lags and ensuring that vehicle alerts are synchronized with real-time traffic conditions.

[0045] 3. Enhance the explainability and interactive experience of traffic violation alerts.

[0046] Based on the semantic understanding capabilities of traffic regulations knowledge graphs, the voice prompts generated by this invention can automatically associate with legal provisions and support drivers in querying the reasons for violations using natural language. Compared to traditional command-based prompts, this significantly improves drivers' understanding of the prompts, reduces operational confusion, and makes information interaction during driving more human-centered.

[0047] 4. Optimize the efficiency and reliability of in-vehicle edge computing.

[0048] By co-optimizing a lightweight multimodal model with onboard computing resources, this invention achieves high-efficiency end-to-end inference performance, ensuring low-latency response in mobile scenarios. Simultaneously, the cross-validation mechanism of multimodal data in current traffic scenarios reduces the risk of misjudgment caused by anomalies in a single sensor, improving the stability of this invention under complex conditions and avoiding the limitations of a single technical path in traditional solutions. Attached Figure Description

[0049] Figure 1 This is a schematic diagram of a traffic violation reminder method based on a multimodal large model, provided as an embodiment of the present invention.

[0050] Figure 2 This is a flowchart illustrating an optional specific implementation method for training a multimodal large model in an embodiment of the present invention.

[0051] Figure 3 This is a flowchart illustrating an optional specific implementation method for the multimodal data preprocessing step in the current traffic scenario in an embodiment of the present invention.

[0052] Figure 4 This is a flowchart illustrating an optional specific implementation method of step S2 in an embodiment of the present invention.

[0053] Figure 5 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0054] To enable those skilled in the art to better understand the technical solutions of the present invention, exemplary embodiments of the present invention are described below in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0055] Where there is no conflict, the various embodiments of the present invention and the features thereof may be combined with each other.

[0056] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0057] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded. Terms such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0058] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having the meaning consistent with their meaning in the context of the relevant art and the invention, and will not be interpreted as having an idealized or overly formal meaning unless expressly so defined herein.

[0059] In the technical solution of this invention, the collection, storage, use, processing, transmission, provision, and disclosure of user personal information all comply with relevant laws and regulations and do not violate public order and good morals. The use of user data in this technical solution follows relevant national laws and regulations (e.g., the "Information Security Technology - Personal Information Security Specification"). For example: appropriate measures are taken for personal information access control; restrictions are imposed on the display of personal information; the purpose of using personal information does not exceed the scope of direct or reasonable association; and explicit identity targeting is eliminated when using personal information to avoid precisely locating a specific individual.

[0060] Definitions of abbreviations and key terms:

[0061] Multi-modal Large Model (MLLM): refers to an artificial intelligence model that can simultaneously process multiple modalities of data such as text, speech, images, and video, and achieve cross-modal semantic understanding and generation through deep learning algorithms.

[0062] Traffic regulations are normative documents formulated or recognized by the state that stipulate the rights, obligations, and legal responsibilities of road traffic participants. In this invention, traffic regulations mainly involve road traffic rules, observance of traffic signals, and prohibited behaviors.

[0063] Real-time inference refers to the process by which a model immediately performs calculations and outputs results after receiving real-time input data, requiring low latency (e.g., ≤500ms) in traffic scenarios.

[0064] The Transformer architecture is a neural network model based on a self-attention mechanism.

[0065] One relevant technology is a voice alert solution based on GPS and fixed speed camera data. This solution obtains vehicle location information through in-vehicle GPS and matches it with pre-stored coordinate data of fixed speed cameras, red-light violation monitors, etc., triggering a voice alert when the vehicle approaches. For example, when the navigation system is 500 meters away from a fixed speed camera, it announces, "Speed ​​camera 500 meters ahead, speed limit 60 km / h." The technical process is: GPS positioning → coordinate matching to the speed camera database → triggering alert logic → voice announcement. The disadvantages of this solution are: 1. Insufficient scenario coverage: It can only provide alerts for pre-stored fixed locations and cannot handle dynamic traffic violation scenarios (such as no-entry signs on temporary construction sections or on-site enforcement instructions from traffic police at non-fixed locations). This is because it lacks the ability to perceive the real-time environment and relies on static database updates. 2. Rigid rules: It cannot adjust the alert strategy according to specific traffic situations. For example, when the road is slippery in rainy weather, it still uses the conventional speed limit alert, failing to consider the impact of environmental factors on safe speed, as it does not integrate environmental data (such as weather and road conditions) for dynamic analysis. 3. Delayed Data Updates: The electronic eye database requires manual maintenance, and newly added or removed monitoring points are difficult to synchronize in a timely manner, leading to inaccurate alerts. For example, if a speed limit sign on a certain road section has been updated, but the database has not been synchronized, the old rules will still be applied to alerts.

[0066] The second related technology is a traffic sign detection scheme based on single-modal image recognition. This scheme uses an onboard camera to capture images of the road ahead, employs a convolutional neural network (CNN) to recognize traffic signs (such as speed limit signs and no-left-turn signs), and triggers voice prompts. For example, when a "speed limit 80" sign is recognized, it announces "Speed ​​limit 80 km / h ahead." The technical process is: camera image acquisition → image preprocessing → sign recognition algorithm → voice generation. The disadvantages of this scheme are: 1. Weak anti-interference capability: In complex lighting conditions (such as backlighting or heavy rain) or occlusion scenarios, the image recognition accuracy drops significantly, easily leading to missed or false alarms. For example, when leaves partially obscure a speed limit sign, the model may misidentify it as another value, as the single-modal image information lacks redundancy and cannot be supplemented and verified by other modalities. 2. Limited semantic understanding: It can only recognize the visual features of the sign and cannot understand the contextual semantics of the sign or the deeper logic of traffic rules. For example, it recognizes a "no parking" sign but cannot determine whether the current road segment is a temporary permitted parking period, due to a lack of understanding and reasoning ability regarding the textual knowledge of traffic rules. 3. Limited Interaction: It can only passively recognize images and cannot engage in multi-round dialogues with the driver or explain the basis for reminders. It is difficult to meet the driver's need for explanation of rules. For example, when the driver asks "Why is it prohibited to turn left?", it cannot provide references to the relevant legal provisions.

[0067] In summary, the traffic violation monitoring and alert technologies involved in the relevant technologies still have many problems and cannot solve the pain point of how to achieve accurate identification and intelligent alerts for traffic violations.

[0068] With the emergence of multimodal large-scale models, integrating text, image, voice, and video data allows for comprehensive perception of the current vehicle's driving status, driver behavior, dynamics of surrounding traffic participants, and traffic signs and environmental information, providing drivers with more accurate and timely alerts. This invention, through deep collaboration between multimodal large-scale models and onboard sensors, achieves accurate identification and intelligent alerts for traffic violations, significantly optimizing the onboard system's environmental perception, real-time decision-making, and human-computer interaction capabilities.

[0069] like Figure 1 As shown, an embodiment of the present invention provides a method for reminding users of traffic violations based on a multimodal large model, which includes:

[0070] Step S1: Obtain multimodal data of the current traffic scene, which includes image data, audio data, and text data.

[0071] Step S2: Input the multimodal data of the current traffic scenario into the pre-trained multimodal large model, and the pre-trained multimodal large model performs inference in real time and outputs the inference results;

[0072] Step S3: Generate a traffic violation reminder message based on the reasoning result.

[0073] In some embodiments, the method further includes: step S0, training a multimodal large model; such as Figure 2 As shown, step S0, the training of the multimodal large model, includes:

[0074] Step S01: Obtain multimodal sample data from multiple different traffic scenarios. The model pre-training stage requires the use of massive traffic scenario multimodal data (image-audio-text triples).

[0075] Step S02: Input the multimodal sample data from the multiple different traffic scenarios into a preset multimodal large model for training;

[0076] Step S03: Introduce real violation cases into the preset multimodal large model for fine-tuning to obtain the pre-trained multimodal large model. Introducing real violation cases for fine-tuning can enhance the model's generalization ability.

[0077] In some embodiments, the architecture of the preset multimodal large model adopts an encoder-decoder structure, wherein the encoder includes: a visual Transformer for processing image data, a speech Transformer for processing audio data, and a text Transformer for processing text data; the preset multimodal large model achieves feature fusion through a cross-modal attention mechanism.

[0078] In some embodiments, the image data includes: road images captured by an in-vehicle camera; the audio data includes: in-vehicle sound data and / or external ambient sound data, in-vehicle sounds such as driver's voice, in-vehicle voice commands, emergency calls, and external sounds such as horns, ambulance / police sirens, construction site commands, and ambient sounds of severe weather (heavy rain, strong winds); the text data includes: a traffic regulation knowledge graph, location and status data; the traffic regulation knowledge graph includes real-time synchronized temporary traffic control information (such as temporary speed limits and road closure notices).

[0079] Specifically, the image data includes: road front images collected by the vehicle-mounted front-view camera (resolution ≥1920×1080), and vehicle surrounding environment monitoring data collected synchronously by the surround-view camera; the positioning and status data specifically include: real-time location and speed data obtained by the GPS / BeiDou dual-mode positioning module, and vehicle steering, braking and other status data read by the OBD interface.

[0080] In some embodiments, step S2, which involves inputting the multimodal data of the current traffic scenario into a pre-trained multimodal large model, and the pre-trained multimodal large model performing real-time inference and outputting inference results, includes the following step S1a: a multimodal data preprocessing step for the current traffic scenario.

[0081] like Figure 3 As shown, step S1a, the multimodal data preprocessing step for the current traffic scenario, includes:

[0082] Step S1a1: Denoise and distortion correction are performed on the image data;

[0083] Step S1a2: Noise reduction and sound source localization are performed on the audio data;

[0084] Step S1a3: Match the positioning and status data with the high-precision map to obtain basic road segment information (such as speed limit and lane attributes).

[0085] In some embodiments, such as Figure 4 As shown, step S2, which involves inputting the multimodal data of the current traffic scenario into a pre-trained multimodal large model, and the pre-trained multimodal large model performing real-time inference and outputting the inference results, includes the following steps:

[0086] Step 21: The pre-trained multimodal large model identifies at least one of the following through image data: traffic signs (such as speed limits and no-turn markings), traffic light status, road markings, and temporary construction signs, and outputs visual semantic labels (such as "speed limit 60km / h" and "no left turn").

[0087] Step 22: The pre-trained multimodal large model identifies speech features through audio data. Specific speech features include: ambulance / police siren sounds indicating a request for vehicles to give way, driver's tired yawn sounds indicating driver fatigue, and rapid sirens indicating impending danger.

[0088] Step 23: The pre-trained multimodal large model performs fusion reasoning based on visual semantic labels, speech features, localization and state data, and traffic regulations knowledge graph, and outputs reasoning results including: violation type and the mapping relationship between violation type and regulation clause;

[0089] The specific inference results include:

[0090] When the current state is determined to be red based on visual semantic labels, and the location and status data identify no signs of braking at the current vehicle speed, the model determines the vehicle's violation type as "risk of running a red light" and establishes a mapping relationship between "risk of running a red light" and legal provisions.

[0091] When the vehicle speed (based on positioning and status data obtained through the positioning module) exceeds the current road segment speed limit (based on the visual semantic label of the speed limit obtained by recognizing the speed limit sign through image data), the model determines the vehicle's violation type as "speeding violation" and establishes a mapping relationship between "speeding violation" and legal provisions.

[0092] When the model detects an ambulance / police siren based on voice features and identifies a vehicle that failed to take evasive action based on location and status data, it determines the vehicle's violation type as "failure to yield to a risk" and establishes a mapping relationship between "failure to yield to a risk" and regulatory clauses.

[0093] This invention utilizes multimodal data from current traffic scenarios to achieve cross-validation and complementary mechanisms, significantly improving the reliability of vehicle violation type identification. For example, the speed limit information for the current road segment can be obtained from visual semantic labels identified from image data, and the speed limit information for the current road segment can be obtained by matching the vehicle's location and status data with a high-precision map; these two speed limit information can be cross-validated. In partially occluded traffic sign recognition scenarios, when the speed limit information for the current road segment cannot be obtained from visual semantic labels, the speed limit information for the current road segment can be obtained by matching the vehicle's location and status data with a high-precision map. Therefore, this invention uses multimodal data from current traffic scenarios to reduce the risk of misjudgment caused by single sensor anomalies, improves the stability of this invention under complex working conditions, and avoids the limitations of single technical paths in traditional solutions.

[0094] This invention enables a dynamic adjustment mechanism using multimodal data from current traffic scenarios, allowing it to handle dynamic traffic violation scenarios. For example, when rain or snow is detected based on ambient sound data from the audio data, the speed limit reminder is adjusted to the rain or snow speed limit; when temporary traffic control information (such as temporary speed limits, road closure signs, no-entry signs for temporary construction sections, and on-site enforcement instructions from traffic police at non-fixed locations) is detected based on image data, the temporary traffic control or on-site enforcement is treated as a higher priority traffic violation reminder.

[0095] In some embodiments, step S3, generating a traffic violation reminder based on the reasoning result, includes:

[0096] Personalized reminders are generated based on the type of violation (such as speeding, running a red light, or not driving in the correct lane), its severity, and the driving scenario. The severity includes the severity of speeding determined by the proportion of speeding exceeded and / or the severity of the violation determined by traffic regulations. The driving scenario includes information such as whether the vehicle is located on urban roads or highways, obtained by matching the location and status data with a high-precision map.

[0097] The personalized reminder content will be sent to the vehicle's speakers for playback; for example, a speeding reminder: "Current speed is 85 km / h, the speed limit ahead is 60 km / h, please slow down." TTS (text-to-speech) technology can be used to support multilingual and multi-voice broadcasts, and the speech rate and tone can be adjusted according to the urgency (e.g., the speech rate increases and the tone rises when there is an emergency violation).

[0098] In some embodiments, the method further includes: step 4, a voice query step; step 4, the voice query step includes:

[0099] When a voice query request is received, the system outputs the corresponding legal clause explanation information based on the mapping relationship between violation type and legal clause. For example, if the voice query is "Why am I being reminded of a violation?", the system outputs the explanation information "You are turning left in a prohibited left-turn lane, violating Article 35 of the Road Traffic Safety Law regarding the separation of vehicles into lanes".

[0100] In some embodiments, the pre-trained multimodal large model is a lightweight multimodal large model (with ≤10 billion model parameters, compressed through knowledge distillation and quantization) deployed on an in-vehicle terminal, combined with GPU / TPU acceleration chips to achieve an end-to-end inference latency of ≤300ms.

[0101] In some embodiments, high-frequency traffic regulations are cached locally on the vehicle terminal, and temporary traffic control information (such as temporary speed limits and road closure notices) is synchronized in real time via 4G / 5G networks to ensure the timeliness of the reminder rules.

[0102] Based on the same inventive concept, embodiments of the present invention also provide a traffic violation reminder system based on a multimodal large model, configured to implement any of the methods described in the above embodiments, the system comprising:

[0103] The acquisition module is used to acquire multimodal data of the current traffic scene, which includes image data, audio data, and text data.

[0104] The model processing module is used to input the multimodal data of the current traffic scenario into a pre-trained multimodal large model, and the pre-trained multimodal large model performs inference in real time and outputs the inference results.

[0105] The reminder module is used to generate traffic violation reminder information based on the reasoning results.

[0106] Based on the same inventive concept, embodiments of the present invention also provide an electronic device. Figure 5 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Figure 5As shown, an embodiment of the present invention provides an electronic device including: one or more processors 101, a memory 102, and one or more I / O interfaces 103. The memory 102 stores one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement any of the methods described in the above embodiments; the one or more I / O interfaces 103 are connected between the processor and the memory, configured to enable information interaction between the processor and the memory.

[0107] The processor 101 is a device with data processing capabilities, including but not limited to a central processing unit (CPU); the memory 102 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), and flash memory (FLASH); the I / O interface (read / write interface) 103 is connected between the processor 101 and the memory 102, and can realize information interaction between the processor 101 and the memory 102, including but not limited to a data bus (Bus).

[0108] In some embodiments, the processor 101, memory 102, and I / O interface 103 are interconnected via bus 104, and thus connected to other components of the computing device.

[0109] In some embodiments, the one or more processors 101 include a field-programmable gate array.

[0110] Based on the same inventive concept, embodiments of the present invention also provide a computer-readable medium. This computer-readable medium stores a computer program, wherein, when executed by a processor, the program implements the steps of any of the methods described in the above embodiments. The computer-readable storage medium may be a volatile or non-volatile computer-readable storage medium.

[0111] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software can be distributed on a computer-readable storage medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

[0112] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable program instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0113] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0114] The computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of the invention.

[0115] The computer program product described herein can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0116] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0117] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0118] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0119] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0120] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in conjunction with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in conjunction with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of the invention as set forth in the appended claims.

Claims

1. A method for reminding users of traffic violations based on a multimodal large model, characterized in that, include: Acquire multimodal data of the current traffic scene, which includes image data, audio data, and text data; The multimodal data of the current traffic scenario is input into a pre-trained multimodal large model, which then performs real-time inference and outputs the inference results. Traffic violation alerts are generated based on the reasoning results.

2. The method according to claim 1, characterized in that, Also includes: Steps for training a multimodal large model; The steps for training the multimodal large model include: Acquire multimodal sample data from multiple different traffic scenarios; The multimodal sample data from the various traffic scenarios are input into a pre-defined multimodal large model for training. Real-world violation cases are introduced into a pre-set multimodal large model for fine-tuning to obtain the pre-trained multimodal large model; The pre-defined multimodal large model architecture adopts an encoder-decoder structure, wherein the encoder includes: a visual Transformer for processing image data, a speech Transformer for processing audio data, and a text Transformer for processing text data; the pre-defined multimodal large model achieves feature fusion through a cross-modal attention mechanism.

3. The method according to claim 1, characterized in that, The image data includes: road images captured by vehicle-mounted cameras; the audio data includes: in-vehicle sound data and / or external ambient sound data; the text data includes: traffic regulations knowledge graph, location and status data.

4. The method according to claim 3, characterized in that, Before the step of inputting the multimodal data of the current traffic scenario into a pre-trained multimodal large model, and the pre-trained multimodal large model performing real-time inference and outputting inference results, the following steps are included: a multimodal data preprocessing step for the current traffic scenario; the multimodal data preprocessing step for the current traffic scenario includes: The image data is denoised and distortion corrected. The audio data is subjected to noise reduction and sound source localization. The location and status data are matched with a high-precision map to obtain basic road segment information.

5. The method according to claim 1, 2 or 3, characterized in that, The step of inputting the multimodal data of the current traffic scenario into a pre-trained multimodal large model, and the pre-trained multimodal large model performing real-time inference and outputting inference results includes: The pre-trained multimodal large model identifies at least one of traffic signs, traffic light status, road markings, and temporary construction signs through image data and outputs visual semantic labels; The pre-trained multimodal large model identifies speech features through audio data; The pre-trained multimodal large model performs fusion reasoning based on visual semantic labels, speech features, localization and state data, and traffic regulations knowledge graph. The output reasoning results include: violation type and the mapping relationship between violation type and regulation clause.

6. The method according to claim 1, 2 or 3, characterized in that, The step of generating traffic violation reminder information based on the reasoning result includes: Personalized reminders are generated based on the type and severity of the violation and the driving scenario. The personalized reminder will be sent to the car's speakers for playback.

7. The method according to claim 1, 2 or 3, characterized in that, Also includes: Voice query steps; The voice query steps include: When a voice query request is received, the system outputs the explanation of the corresponding legal clause based on the mapping relationship between violation type and legal clause.

8. A traffic violation alert system based on a multimodal large model, characterized in that, The system, configured to implement the method as described in any one of claims 1 to 7, comprises: The acquisition module is used to acquire multimodal data of the current traffic scene, which includes image data, audio data, and text data. The model processing module is used to input the multimodal data of the current traffic scenario into a pre-trained multimodal large model, and the pre-trained multimodal large model performs inference in real time and outputs the inference results. The reminder module is used to generate traffic violation reminder information based on the reasoning results.

9. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1 to 7.

10. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images