Traffic illegal behavior identification method and device

Abstract

The invention discloses a traffic illegal behavior identification method and device. The method comprises the following steps: acquiring a plurality of to-be-recognized images; performing target detection on the plurality of to-be-recognized images to obtain a detection result; adopting the target object and the plurality of traffic elements to fill the preset text cue word to obtain a cue instruction set; and processing the prompt instruction set and the plurality of to-be-identified images by adopting the target generative model to obtain an illegal judgment text. According to the method and the device, the technical problem that the false alarm rate is high due to the fact that a recognition method for illegal behaviors such as red light running depends on single visual detection or a rule engine and lacks semantic understanding and multi-mode cooperative reasoning ability for traffic elements in the prior art is solved.

Description

Technical Field

[0001] This application relates to the field of intelligent transportation technology, and more specifically, to a method and apparatus for identifying traffic violations. Background Technology

[0002] Current urban traffic violation recognition systems, especially those for identifying red-light running by motor vehicles, face a series of technical bottlenecks and challenges. However, the methods used by these technologies to identify violations such as red-light running rely on single visual detection or rule engines, such as using geomagnetic induction, video contour detection, or fixed thresholds to determine whether a vehicle has crossed the stop line. These methods lack semantic understanding of traffic elements and multimodal collaborative reasoning capabilities, resulting in high false alarm rates and an inability to distinguish between legal lane changes and illegal red-light running.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This application provides a method and apparatus for identifying traffic violations, which at least solves the technical problem that the identification methods for violations such as running red lights used in related technologies rely on a single visual detection or rule engine, lack semantic understanding of traffic elements and multimodal collaborative reasoning capabilities, resulting in a high false alarm rate.

[0005] According to one aspect of the embodiments of this application, a method for identifying traffic violations is provided, comprising: acquiring multiple images to be identified, wherein the multiple images to be identified include a time-continuous image sequence of a target object in a traffic scene; performing target detection on the multiple images to be identified to obtain detection results, wherein the detection results include the movement trajectory of the target object and multiple traffic elements in the traffic scene; filling preset text prompt words with the target object and multiple traffic elements to obtain a prompt instruction set, wherein the prompt instructions in the prompt instruction set are used to guide a target generative model to analyze the traffic elements in the images to be identified according to a preset logical order; and processing the prompt instruction set and the multiple images to be identified using the target generative model to obtain a violation judgment text, wherein the target generative model is used to capture the semantic association between multiple traffic elements, and the violation judgment text is used to determine the traffic violation of the target object in the traffic scene.

[0006] In this embodiment, a multimodal perception fusion approach is adopted. By collecting and analyzing the time-continuous image sequence of the target object in a traffic scene, and combining the target detection algorithm to track the target object's movement trajectory and key elements in the traffic scene, the detected multimodal information is used to fill in preset text instructions to form a guiding analysis framework. This guides the target generative model to deeply understand the image content in a logical order, capture the deep semantic relationships between traffic elements, and generate violation judgment text describing the target object's traffic violation behavior. This achieves the goal of accurately identifying and understanding the traffic scene, thereby realizing the technical effects of automating and intelligently judging violations under complex conditions and significantly reducing the false alarm rate. It also solves the technical problem that the identification methods for violations such as running red lights used in related technologies rely on a single visual detection or rule engine, lack semantic understanding of traffic elements and multimodal collaborative reasoning capabilities, resulting in a high false alarm rate. Attached Figure Description

[0007] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0008] Figure 1 This is a hardware structure block diagram of a computer terminal for a method of identifying traffic violations according to an embodiment of this application;

[0009] Figure 2 This is a flowchart of a method for identifying traffic violations according to an embodiment of this application;

[0010] Figure 3 This is a schematic diagram of the overall process of a method for identifying traffic violations according to an embodiment of this application;

[0011] Figure 4 This is a schematic diagram of motion trajectory tracking for a method of identifying traffic violations according to an embodiment of this application;

[0012] Figure 5 This is a traffic element detection diagram of a method for identifying traffic violations according to an embodiment of this application;

[0013] Figure 6 This is a flowchart of the model training process for a method for identifying traffic violations according to an embodiment of this application.

[0014] Figure 7 This is a schematic diagram of reinforcement learning training for a traffic violation identification method according to an embodiment of this application;

[0015] Figure 8This is a structured report generation flowchart of a method for identifying traffic violations according to an embodiment of this application;

[0016] Figure 9 This is a schematic diagram of the structure of a traffic violation identification device according to an embodiment of this application. Detailed Implementation

[0017] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0018] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0019] Current urban traffic violation recognition systems primarily rely on a combination of traditional image processing and rule engines. For example, they use methods like geomagnetic induction, video contour detection, or fixed thresholds to determine if a vehicle has crossed the stop line. While these methods are practical in standardized intersection environments, they face significant limitations in complex real-world scenarios: First, traffic light status recognition generally uses binary classification models, making it difficult to distinguish the prohibited direction of a red light (e.g., a left-turn red light while straight-ahead traffic is permitted), and to address misjudgments caused by multi-phase signal light timing changes and occlusion. Second, they lack semantic-level understanding of the spatial relationship between vehicle trajectories and lane lines, making it difficult to effectively distinguish between "briefly crossing the line to avoid an emergency vehicle" and "deliberately running a red light." Third, existing systems are mostly based on a serial architecture of independent modules, with information fragmented between sub-tasks (object detection, trajectory tracking, signal recognition), making it difficult to form a collaborative reasoning mechanism, resulting in high false alarm rates and weak evidence chains. To address these technical problems, this application provides corresponding solutions, which are detailed below.

[0020] The traffic violation identification method provided in this application can be executed on a mobile terminal, computer terminal, or similar computing device. Figure 1 A hardware block diagram of a computer terminal for implementing a method to identify traffic violations is shown. Figure 1 As shown, the computer terminal 10 may include one or more processors (shown as 102a, 102b, ..., 102n in the figure) (the processor may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission module 106 for communication functions connected via wired and / or wireless networks. In addition, it may also include: a display, a keyboard, a cursor control device, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, and a BUS bus. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0021] It should be noted here that, in some optional embodiments, the above... Figure 1 The computer terminal shown may include hardware elements (including circuitry), software elements (including computer code stored on a computer-readable medium), or a combination of both hardware and software elements. It should be noted that... Figure 1 This is only one instance of a specific particular instance, and is intended to illustrate the types of components that may exist in the aforementioned computer terminal.

[0022] In the above operating environment, the embodiments of this application provide an embodiment of a method for identifying traffic violations. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0023] Figure 2 This is a flowchart of a method for identifying traffic violations according to an embodiment of this application, such as... Figure 2 As shown, the method includes the following steps:

[0024] Step S202: Obtain multiple images to be identified, wherein the multiple images to be identified include a time-continuous image sequence of the target object in a traffic scene.

[0025] In step S202 above, the target object refers to the entity in a traffic scene that needs to be identified and judged to have committed illegal acts, including but not limited to motor vehicles, pedestrians, or other means of transportation. The traffic scene refers to the real environment in which vehicles and pedestrians participate in traffic activities, such as a traffic environment with traffic light control, clearly defined lane lines, and zebra crossings. A time-continuous image sequence refers to a series of images continuously captured by a camera. These images are closely connected on the timeline and can reflect the dynamic behavior of the target object in the traffic scene. The time-continuous image sequence is used to capture the entire process of a vehicle passing through an intersection, including key actions such as approaching traffic lights, crossing the stop line, and leaving the intersection.

[0026] Step S204: Target detection is performed on multiple images to be identified to obtain detection results, which include the movement trajectory of the target object and multiple traffic elements in the traffic scene.

[0027] In step S204 above, target detection is mainly used to capture moving targets such as motor vehicles, pedestrians, and non-motorized vehicles in traffic scenes in real time, as well as static traffic elements such as traffic lights, zebra crossings, and lane lines. It should be noted that the movement trajectory of a target object refers to the path and direction of the target object's movement over a period of time, which can be composed of a series of location points. For example, in an intelligent traffic monitoring system, the movement trajectory helps the system understand how vehicles cross intersections, especially their relationship with traffic light signals and stop lines, in order to determine whether vehicles or other target objects have committed traffic violations such as running red lights (the following explanation will use running red lights as an example).

[0028] In some embodiments of this application, real-time or near-real-time target detection can be performed on image sequences to ensure accurate capture of target objects and traffic elements from each frame. Combining the target detection results with license plate recognition technology, multi-level matching is used to gradually lock onto the target vehicle and track its entire movement trajectory. Simultaneously, image description models and multimodal semantic models are employed for verification to ensure the accuracy of traffic element recognition, particularly for partially occluded or complex-shaped traffic elements, improving the system's robustness to multi-source information fusion processing. Through multi-image information integration and multi-level verification, the system can accurately depict the trajectory of the target object in multiple consecutive images. Even brief visual occlusions or blurred license plates will not affect the complete capture and judgment of its violations.

[0029] In some embodiments of this application, the target object includes the target vehicle in the image to be identified, which refers to the motor vehicle selected by the monitoring system as the object of analysis, such as a vehicle suspected of traffic violations.

[0030] To accurately describe the movement trajectory of the target object, the detection result can be obtained in the following way: Detect the first image to be identified in the time-continuous image sequence to obtain multiple candidate vehicle images corresponding to multiple candidate vehicles, wherein the first image to be identified includes the first frame image in the time-continuous image sequence; perform license plate detection on the multiple candidate vehicle images to obtain the target vehicle image that matches the preset license plate; determine the first visual semantic information corresponding to the target vehicle image; determine the first detection result corresponding to the target vehicle based on the first visual semantic information, wherein the first detection result includes the driving trajectory of the target vehicle.

[0031] Specifically, visual semantic information refers to high-level information extracted from images that includes the type, position, posture, and relationship with other objects. Visual semantic information may include, but is not limited to, the vehicle's direction of travel, its distance from the stop line or other traffic elements, and the vehicle's speed and acceleration.

[0032] To overcome the problems of license plate obstruction and reading difficulties that may occur during vehicle operation, the first detection result corresponding to the target vehicle can be determined as follows: The license plate information and second visual semantic information of the vehicle in a second image to be identified in a time-continuous image sequence are determined. The second image to be identified includes any image to be identified in the time-continuous image sequence other than the first image to be identified. A first similarity score is determined based on the license plate information and a preset license plate, and the first visual semantic information and the second visual semantic information. The first similarity score quantifies the degree of similarity between the vehicle and the target vehicle, and the ratio of the similarity weights between the license plate information and the second visual semantic information corresponding to the second image to be identified decreases as the time sequence increases. The driving trajectory of the target vehicle in the time-continuous image sequence is determined based on the first similarity score, thus obtaining the first detection result.

[0033] Specifically, the first visual semantic information refers to the semantic information of the target vehicle extracted from the first image to be identified, including the vehicle's type, location, direction of travel, and relative relationship with traffic elements, used to initially locate the vehicle's state in the traffic scene. The second visual semantic information corresponds to the information of the target vehicle in another frame of the temporally continuous image sequence besides the first image to be identified. It also includes the vehicle's location, direction, and relationship with the surrounding environment, but focuses more on capturing details of the vehicle's dynamic changes. The first similarity score is used to evaluate the degree of matching between the vehicle information in the second image to be identified and the target vehicle's information in the first image to be identified, comprehensively considering both direct matching of license plate information and indirect similarity of visual semantic information.

[0034] To facilitate understanding of the process for determining the first detection result, the following explanation is provided in conjunction with some specific embodiments. Specifically, the steps for locating and tracking the target license plate vehicle and capturing the vehicle's movement trajectory include:

[0035] Given a vehicle license plate Q, we can locate vehicles with that license plate in all images (images to be identified) to perform target vehicle trajectory tracking across multiple images. For example, a multi-level matching strategy can be used. First, the first input image is often an image of a vehicle at a crosswalk without running a red light. In this case, the vehicle is usually facing the camera. In this image, a vehicle detection model is used to detect all vehicles, obtaining candidate vehicle groups {C1, C2, ..., Cm}. The detected local vehicle bounding boxes are cropped and enlarged, and a license plate detection model is used to detect and match the license plate to obtain the target vehicle Ct. Finally, the SIGLIP visual model is used to extract visual semantic information from the target vehicle image to obtain... As a candidate vehicle.

[0036] In the second step, all images except the first are used for matching. Because the vehicle is in motion, the second image captures the vehicle near the zebra crossing with significant obstruction of the license plate. The third image shows the vehicle leaving the zebra crossing, exhibiting a tendency to turn left or right. Since the vehicle is not directly facing the camera, the license plate matching success rate gradually decreases. Therefore, a multi-level matching method can be used to track the target vehicle. As time progresses, the similarity ratio of visual semantic matching gradually increases, while the similarity ratio of license plate matching decreases. The formula for calculating similarity is as follows:

[0037]

[0038] Here, similarity is the first similarity score. Let be the feature vector of the target vehicle at time t. Let be the feature vector of any candidate vehicle, and total be the total number of frames in the video sequence. Sim(plate) is the similarity score between the visual feature vectors of the target vehicle and the candidate vehicle by the visual base model, and Sim(plate) is the similarity score between the license plate of the target vehicle and the license plate of the candidate vehicle by the license plate matching algorithm.

[0039] Figure 4 This is a schematic diagram of motion trajectory tracking for a method of identifying traffic violations according to an embodiment of this application, as shown below. Figure 4 As shown, in some embodiments of this application, the motion trajectory can be determined through the following steps:

[0040] (1) Figure 1 Vehicle localization and feature extraction.

[0041] First, vehicle inspection: Figure 1 The first frame (in a temporally continuous image sequence) is input into a pre-trained general vehicle detection model, such as YOLOv5. The goal is to detect all vehicles in the scene, generating multiple local images containing vehicle bounding boxes. These local images form the basis for subsequent feature analysis and tracking. Next, license plate localization and target region determination: The aforementioned local images are analyzed a second time using a specialized license plate detection model, such as a deep learning-based license plate localization algorithm, to determine the specific target vehicle. The license plate is considered the unique identifier of the vehicle; recognizing the license plate ensures the accuracy of subsequent tracking. Then, visual feature extraction: After selecting the target vehicle, the SIGLIP (Self-Attention Interactive Feature Encoder) model is used to perform in-depth analysis of the region where the target vehicle is located, extracting its visual features. These features include not only the vehicle's appearance details but also its relative position to the surrounding environment and its pose, providing crucial information for subsequent multi-image matching and understanding.

[0042] (2) Figure 2 And subsequent images: multi-image matching and feature analysis.

[0043] First, vehicle detection and feature acquisition: Figure 2 (i.e., the second frame in a temporally continuous image sequence) and all subsequent frames are subjected to AND operations. Figure 1 The same vehicle detection and local map generation steps are followed. Then, the local map of each frame is fed into the SIGLIP and license plate matching models respectively to obtain candidate region features and target tracking region feature information, which are used to capture changes in the vehicle's position, orientation, and behavior at different time points. Next, feature comparison and matching: Figure 2 Characteristics of candidate regions and Figure 1 The system compares the target region features and calculates their similarity scores. It's worth noting that the Siglip model excels at handling visual semantic information, thus effectively improving the accuracy and robustness of multi-image matching, especially when the license plate is obscured or unclear. Next, trajectory tracking: through a multi-level matching strategy (combining license plate information and visual semantic features), the system can determine the exact location of the target vehicle at different time points, thereby constructing the vehicle's trajectory. As the image sequence progresses, the importance of visual semantic matching gradually increases, while the proportion of license plate matching gradually decreases, to adapt to the uncertainties brought about by vehicle movement and environmental changes.

[0044] (3) Comprehensive analysis and trajectory determination.

[0045] First, the judgment of trajectory continuity: comprehensive Figure 1 , Figure 2Including feature information from all subsequent images, the system employs dynamic prompt word engineering and a decoder network to progressively deduce the target vehicle's movement path. By analyzing the target vehicle's positional changes within each frame and its dynamic relationship with traffic elements (such as traffic lights, zebra crossings, and lane markings), the system can construct a coherent and accurate vehicle trajectory. Secondly, multimodal fusion reasoning: A multimodal large language model (i.e., a target generative model) is introduced for in-depth analysis, combining the visual features of each frame with the model's text understanding capabilities to progressively understand and judge whether the vehicle's behavior complies with traffic rules, especially in the interaction between the vehicle and traffic signals. Then, result output and verification: Finally, based on the feature and correlation analysis of all images, the system generates a detailed vehicle trajectory report, which will serve as an important basis for subsequent judgments of violations.

[0046] In addition to object detection, traffic element detection is also required: a visual detection model is used to detect multiple images to be identified, resulting in multiple candidate bounding boxes containing candidate traffic elements; descriptive text is determined for the image region corresponding to each candidate bounding box, wherein the descriptive text describes the name of the candidate traffic element within the candidate bounding box; semantic matching is performed between the descriptive text and preset traffic element names to obtain a second similarity score, and the second similarity score is compared with a preset threshold to obtain a comparison result; based on the comparison result, a second detection result corresponding to the traffic element is determined.

[0047] Specifically, in some embodiments of this application, in order to achieve accurate positioning of key traffic elements (such as zebra crossings, traffic lights, lane lines, etc.), since these elements often face challenges such as variable shapes, partial occlusion, and complex lighting in real traffic scenarios, single detection methods are prone to missed detections or false detections. Therefore, a multi-path collaborative positioning strategy can be adopted, integrating target detection, image description generation, and multimodal semantic matching technologies. A multi-level verification mechanism can be used to improve the accuracy and robustness of positioning, including:

[0048] First, a general visual detection model is used to perform preliminary analysis on the input image, extracting candidate bounding boxes that may contain the target traffic element, forming a candidate set B = b1, b2, ..., bn. Second, for the local image region cropped from each candidate box, an image description model is used to generate a natural language description. This description is then matched with the standard semantic name of the corresponding traffic element (such as "zebra crossing," "traffic light," etc.) using a multimodal semantic model such as Siglip, calculating their semantic similarity score (second similarity score). Third, a multi-level judgment mechanism is set based on the similarity score: if the matching score of a candidate box exceeds a higher threshold 'a', its location result is considered reliable, and it is directly confirmed as the corresponding traffic element; if the score is lower than 'a' but higher than the intermediate threshold 'b', it indicates that the candidate box has a certain degree of uncertainty.

[0049] Figure 5 This is a traffic element detection diagram of a traffic violation identification method according to an embodiment of this application, as shown in the diagram. Figure 5 As shown, in some embodiments of this application, traffic element detection can be performed in the following ways:

[0050] (1) Image and target prompt words are input into the object detection model to obtain candidate boxes. A single traffic scene image and target prompt words containing the names of specific traffic elements (such as "zebra crossing" and "traffic light") are used as input and fed into the object detection model, such as GroundingDINO. This model is based on deep learning and can detect multiple candidate regions in the image that are related to the prompt words through the guidance of the prompt words.

[0051] (2) Candidate boxes are input into an image description model to obtain image descriptions. Using an image description model, such as CLIP or BLIP, the candidate box regions from the first step are refined and analyzed to generate a natural language description for each candidate box, aiming to enhance the model's ability to understand and distinguish candidate regions. The image description model can provide detailed semantic information for each candidate box, helping subsequent multimodal matching to further confirm the target and avoiding false detections or missed detections that may be caused by single visual detection.

[0052] (3) Calculate similarity scores. Combining the output of the image description model with the target prompt words, the multimodal large language model Siglip is used for semantic matching to calculate the similarity score between each candidate box and the prompt word description. This step solves the semantic gap between visual features and text descriptions. Through multimodal fusion, the robustness and accuracy of traffic element recognition are improved, especially in complex scenes with variable element shapes and partial occlusion.

[0053] (4) Comparison of similarity score with threshold. Two thresholds, a and b, are set to filter candidate boxes. When the similarity score exceeds threshold a, the target corresponding to the candidate box is directly confirmed as the traffic element to be detected. If the score is lower than a but higher than b, the candidate box is considered to have a certain degree of uncertainty and needs further analysis to reduce misjudgment.

[0054] (5) Expansion of Uncertain Boxes and Multimodal Re-detection. Candidate boxes with similarity scores between thresholds a and b are expanded to cover more possible feature information. Then, more detailed location prompts are used to interact with a multimodal large language model to re-evaluate the accuracy of the candidate boxes and return the precise coordinates of the target candidate box. For example, the location prompts could be: "This image is a traffic scene image. Please determine whether there is a zebra crossing / lane line / traffic light in the image. If so, accurately locate the target and return the precise coordinates." <object> (x1,y1)(x2,y2)< / object>The format returns the target candidate box. Here, x1, y1 are the coordinates of the top-left corner, and x2, y2 are the coordinates of the bottom-right corner.

[0055] This mechanism solves the problem of inaccurate feature localization under low-confidence detection. By expanding the region box and multimodal re-detection, the system can increase the observation range of the target. Combined with the guidance of text instructions, the target can be analyzed more meticulously, thereby improving the accuracy of feature localization, especially the recognition ability when the target border does not completely match.

[0056] Step S206: Fill the preset text prompt words with the target object and multiple traffic elements to obtain a prompt instruction set. The prompt instruction set is used to guide the target generative model to analyze the traffic elements in the image to be identified in a preset logical order.

[0057] In step S206 above, the preset text prompts are a set of pre-designed natural language instructions used to guide the multimodal large language model to perform specific tasks. The prompts can cover various levels from target localization and feature recognition to logical reasoning, ensuring that the model output follows a preset logical order. The prompt instruction set is a collection of preset text prompts filled with specific target object and traffic element information. The prompt instruction set is the source of instructions for the model to perform in-depth analysis and reasoning, guiding the model to analyze whether the target vehicle's behavior is illegal according to traffic regulations and logical order.

[0058] In some embodiments of this application, text prompts containing key descriptions and analysis instructions for traffic scenes can be designed, such as "You are a professional traffic assistant. Please locate the lane lines in the image, then determine the status of the traffic lights, and finally analyze whether the target vehicle ran a red light." Then, the target object information detected in the previous steps (such as vehicle location and license plate) and multiple traffic elements (such as traffic light color, lane line direction, and zebra crossing location) are used to fill the placeholders in the preset text prompts, converting them into specific prompt instructions. The filled prompt instructions are then combined into an instruction set, presented sequentially or hierarchically, ensuring that the model can perform analysis and reasoning according to a preset logical order. For example, if the target detection result indicates the presence of traffic lights and zebra crossings (traffic elements) in the image to be identified, the traffic lights and zebra crossings can be used to construct a corresponding instruction set to guide the target generative model in analyzing the traffic lights, zebra crossings, and target vehicles.

[0059] Step S208: The target generative model is used to process the prompt instruction set and multiple images to be identified to obtain the violation judgment text. The target generative model is used to capture the semantic relationship between multiple traffic elements, and the violation judgment text is used to describe the traffic violation behavior of the target object in the traffic scene.

[0060] In step S208 above, the target generative model processes and generates text while understanding visual content, enabling fine-grained semantic association analysis of multiple elements in the traffic scene. The target generative model makes a judgment on whether a violation has occurred by understanding and reasoning about the logical relationship between the state of traffic elements and the behavior of the target vehicle. The violation judgment text is the result of the model's output based on the prompt instruction set and image analysis; it is a structured text report that details the driving situation of the target object (such as a vehicle) and the states of traffic elements in the traffic scene.

[0061] In some embodiments of this application, a set of prompt instructions and multiple images to be identified are input into a target generative model. The model first performs visual encoding on the images, extracts visual features of key traffic elements, and then fuses these features with text information in the instruction set to understand the relationship between visual features and text instructions. Based on the feature mapping network and large language model architecture within the model, multimodal semantic association analysis is performed to capture deep semantic relationships between elements such as traffic light status, lane lines, zebra crossings, and vehicle positions, and to understand and infer whether the behavior of the target vehicle complies with traffic rules. The model outputs a structured text report, which may contain the model's analysis results of the traffic scene. For example, the report may include: "In the first frame image, the vehicle is in front of the zebra crossing, and the red light is on; the second frame image shows that the vehicle begins to cross the line; the traffic light is still red at this time, prohibiting straight ahead; therefore, the vehicle has committed the illegal act of running a red light."

[0062] By employing a goal-oriented generative model, this paper addresses the limitation of single-modal (such as visual or textual) information processing in judging red-light violations. It achieves deep integration of visual and semantic information, improving the accuracy and robustness of the judgment. In some embodiments of this application, the generated violation judgment text not only includes the judgment result but also records the reasoning process in detail, greatly enhancing the transparency and interpretability of the entire system and facilitating subsequent adjudication and manual review.

[0063] In some embodiments of this application, the target generative model is trained in the following manner: An initial target generative model is trained using a first training dataset and a first text prompt to obtain a first target generative model. The first training dataset includes a first historical image with traffic element annotations, and the first text prompt instructs the initial target generative model to locate traffic elements. The first target generative model is used to locate the positions of the first target object and the first traffic element in the first historical image. A second target generative model is trained using a second training dataset and a second text prompt to obtain a second target generative model. The second training dataset includes a second historical image with traffic element annotations and traffic element status annotations. The image and the second text prompt are used at least to instruct the first target generative model to judge the state of traffic elements. The second target generative model is used to judge the state and change trend of the second target object and the second traffic element in the second historical image. The second target generative model is trained using a third training dataset and a third text prompt to obtain a target generative model. The third training dataset includes a third historical image sequence with traffic element annotations, traffic element state annotations, and target object violation annotations. The third text prompt is used at least to instruct the second target generative model to perform temporal continuous reasoning on the target object and traffic elements. The target generative model is used to determine the traffic violation conclusion corresponding to the third target object in the third historical image sequence.

[0064] In the first stage of model training: (1) First training dataset: containing first historical images labeled with traffic elements (such as traffic lights, zebra crossings, lane lines), used to train the initial target generative model to identify the location of these traffic elements. This is the foundational stage for the model to learn visual understanding of traffic scenes. (2) First text prompts: instructions used to guide the model to understand and locate traffic elements, such as "locate the traffic lights in the image". Through training, the model learns to respond to these specific instructions and accurately identify traffic elements, laying the foundation for more complex tasks in the future. (3) First target generative model: the model trained based on the first training dataset and the first text prompts has initially acquired the ability to locate traffic elements. This is the initial stage of building the model's functionality, providing visual localization support for more advanced analysis and reasoning.

[0065] Specifically, using the labeled information in the first training dataset, combined with the first text prompt, such as "locate lane lines in the image," an initial target generative model is trained. Through learning, the model is able to accurately locate the positions of traffic elements such as lane lines in the image.

[0066] In the second stage of model training: (1) Second training dataset: Based on the first training dataset, a second historical image with annotations of traffic element states (such as the current state of traffic lights) was added. This additional annotation information is used to guide the model to learn and understand the dynamic changes of traffic elements. (2) Second text prompts: These instruct the model not only to locate traffic elements but also to judge their states, such as "determine whether the traffic light prohibits straight-through traffic". Through training, the model can judge the state of traffic elements based on visual input. (3) Second target generative model: Based on the first target generative model, it is trained using the second training dataset and the second text prompts. It can identify the state of traffic elements and predict behavioral trends. This is the stage of deepening the model's function and upgrading from static localization to dynamic state judgment.

[0067] Specifically, a second training dataset is used, which includes annotations of traffic element states, such as the current state of traffic lights (green light allows passage, red light prohibits passage), and a second text prompt, such as "determine whether the traffic light prohibits left turns," to further train the first objective generative model, enabling it to understand and judge the dynamic changes of traffic elements.

[0068] In the third stage of model training: (1) Third training dataset: containing a third historical image sequence with traffic element annotations, status annotations, and object violation annotations, used to train the model to perform continuous temporal reasoning to determine whether there is a traffic violation. (2) Third text prompt words: guiding the model to perform continuous temporal reasoning to determine whether the interaction between the object and traffic elements constitutes a violation, such as "based on the image sequence, determine whether the vehicle crossed the stop line when the red light is on". (3) Target generative model: trained through the third training dataset and the third text prompt words, it has the ability to fully understand and judge violations in traffic scenarios and can generate detailed violation analysis reports.

[0069] Specifically, a third training dataset, which contains annotations of the target object's violations, is used in conjunction with a third text prompt, such as "Based on the image sequence, determine whether the vehicle crossed the stop line during the red light," to train the second target generative model. This enables the model to perform continuous temporal reasoning based on multiple consecutive image inputs to determine whether the target object's behavior constitutes a traffic violation.

[0070] Each stage of the training described above builds upon the previous stage, progressing from static localization to dynamic state judgment, and then to continuous time reasoning. This gradually deepens the model's understanding of traffic scenarios, ultimately reaching an advanced stage where it can automatically identify and judge traffic violations.

[0071] In some embodiments of this application, the initial target generative model includes a visual encoder, a feature mapping network, and an initial generative model, wherein the visual encoder is used to convert the input image into local image features, and the feature mapping network is used to spatially align the local image features with the initial generative model.

[0072] The primary responsibility of a visual encoder is to transform input image data into a series of local image features that are easily understood by machines. These features capture key information in the image, such as color, shape, and texture. Feature mapping networks, acting as a bridge between the visual encoder and the generative model, are responsible for transforming the local image features output by the visual encoder into a feature space suitable for processing by the generative model. This ensures effective communication and interaction between visual features and the linguistic features of the generative model, thereby improving the accuracy and efficiency of multimodal understanding.

[0073] The initial generative model is a text-based model that can understand natural language instructions and generate corresponding text responses. In some embodiments of this application, the initial generative model is used to understand traffic-related instructions (such as "locate traffic lights") and generate descriptive text based on the input visual features. It is a basic module for model training and inference.

[0074] To reduce token consumption during traffic element localization, the first target generative model can be obtained as follows: The visual encoder of the initial target generative model determines a feature sequence corresponding to a first historical image, where the feature sequence captures local information from the first historical image; the feature mapping network of the initial target generative model fuses the feature sequence with the text instruction features corresponding to the first text prompt word to obtain a fused feature vector, where the text instruction features embed labeled features to instruct the initial target generative model to locate traffic elements; the initial generative model of the initial target generative model processes the fused feature vector to obtain predicted text, where the predicted text includes the predicted location of the first traffic element determined based on the labeled features; the loss between the predicted location and the traffic element label is determined, and the model parameters of the initial target generative model are adjusted based on the loss to obtain the first target generative model.

[0075] Tokenized features (learnable token features) are special features embedded in text instructions to guide the model in locating specific traffic elements during prediction. Specifically, a first historical image is fed into a visual encoder. The encoder extracts local features from the image through operations such as convolution and pooling, converting them into a series of feature vectors to form a feature sequence. A feature mapping network then fuses the feature sequence generated by the visual encoder with the embedded features of the first text prompt, generating a fused feature vector. For example, using the self-attention mechanism in Transformer, image features are associated with the instruction features of "locate the zebra crossing" in the text, improving the efficiency of the localization task. The fused feature vector is then input into an initial generative model (such as a Transformer-based large language model), which generates predicted text describing the predicted location of the traffic element. For example, based on the fused features, the model answers, "The zebra crossing is located at the bottom center of the image, x1=0.4, y1=0.8, x2=0.6, y2=0.9."

[0076] The difference between the predicted location generated by the model and the location of traffic elements annotated in the first training dataset is calculated. For example, IoU (Intersection over Union) or L1 and L2 distances are used as loss functions to measure the degree of agreement between the prediction and the actual location. Based on the calculated loss, the model parameters are adjusted through backpropagation to optimize the positioning capability.

[0077] It should be noted that, based on the structure of the basic multimodal large language model, this application's embodiment uses dynamic prompt word engineering to insert learnable token features (labeled features) into text commands to replace text features, thereby achieving the localization of traffic elements. This is an improvement on the use of multimodal large language models for visual localization relationships. In related technologies, multimodal large language models output the top-left x-coordinate, y-coordinate, bottom-right x-coordinate, and y-coordinate sequentially in text form. This significantly consumes tokens and introduces more propagation errors. When a single digit is generated incorrectly, due to the generative model's token-by-token prediction paradigm, the error is used as input and passed to the next output, leading to inaccurate localization results and further affecting the model's judgment of the target's state.

[0078] The use of labeled features enables the model to process traffic elements with higher-level semantic features, rather than simply outputting coordinate values. This approach allows the model to more easily learn the state and attributes of traffic elements, such as traffic light colors, arrow directions, and vehicle orientations, rather than just their geometric positions. Furthermore, by learning labeled features, the model can more flexibly adjust its inference strategy when dealing with varying traffic environments (such as different lighting conditions, changing viewpoints, and occlusion), rather than overly relying on specific coordinate output formats. This improves the model's generalization performance across different cities, intersections, and even non-standard traffic scenarios.

[0079] To accurately locate and identify key traffic elements such as traffic lights and zebra crossings, the fused feature vector can be processed through the following steps to obtain the predicted text: An initial generative model is used to extract target marker features from the text instruction features in the fused feature vector, where the target marker features are determined based on placeholders in the text prompts; a decoder is used to decode the target marker features to obtain the predicted location corresponding to the target marker features, which includes the predicted coordinates and prediction confidence of the first traffic element; the initial generative model is then used to replace the placeholders corresponding to the target marker features with the predicted location to obtain the predicted text.

[0080] Target-labeled features refer to features that use special placeholders within text instructions to instruct the model to focus on and locate specific types of traffic elements (such as traffic lights and zebra crossings). These features are designed to be learnable, meaning that the model gradually learns how to correctly interpret these placeholders during training, thereby accurately performing the localization task. Specifically, special markers, such as placeholders {p}, are embedded in the text instruction features to indicate the type of target the model needs to locate. In target generative models, these labeled features are designed as learnable token sequences. Through interaction with the fused feature vector, the model can understand and respond to these labeled instructions, performing the corresponding localization operation.

[0081] A decoder is a network structure specifically designed to decode specific outputs (such as coordinates) from the hidden states of a model. It can be based on the decoder part of a Transformer. In the task of locating traffic elements, the decoder receives the target label features in the fused feature vector and converts them into a numerical representation of predicted coordinates through internal decoding mechanisms (such as using position encoding and attention mechanisms). It should be noted that the predicted location includes not only coordinate information but also a corresponding confidence score to measure the model's confidence level in predicting the location.

[0082] Based on the predicted location generated by the model, the initial generative model replaces the placeholders in the text instructions with the actual predicted coordinates and confidence information, generating a descriptive text, such as "The location of the traffic light is (x1=0.4, y1=0.8, x2=0.6, y2=0.9), with a confidence level of 0.95". This replacement operation ensures that the model's output conforms to a human-readable format, facilitating subsequent manual review or further system processing.

[0083] In some specific embodiments of this application, a target generative model can be constructed based on a visual encoder, a feature mapping network, and a large language model. The image to be identified or a historical image is encoded by the visual encoder to construct local image features. Where v is an m-dimensional feature, this feature sequence is then input into a feature mapping network to change the feature dimension and transform the visual semantic space into a text semantic space to align with the generative model (large language model). Finally, the transformed visual features and text instruction features are concatenated and fed into the large language model for token-by-token output, as shown in the following formula:

[0084]

[0085] in, The output generated by the multimodal large language model at step i. The visual features are transformed by a feature mapping network. V represents the text instruction features, and V represents the original visual feature sequence.

[0086] This application proposes a set of learnable token feature sequences. The following text prompts were constructed for supervised training:

[0087] The text prompt t1 is: "You are a professional traffic assistant. Please locate the position of the lane line {p} and the position of the traffic light {p} in the image.", where {p} is the insertion position of the learnable feature. To train this task, the answer is constructed as a: "The position of the lane line is {po}, and the position of the traffic light is {po}."

[0088] Since large language models first obtain the output token features during the prediction process, and then query the vocabulary database to output specific words, this application embodiment adds specific word positioning in its implementation. <ground>(Target token features) When the model determines that the input prompt word is correct, the corresponding token features can be extracted and fed into the decoder as information for multimodal fusion. The decoder is a lightweight transformer architecture, and the final output coordinates and corresponding confidence scores are as follows:

[0089]

[0090] in, Here, represents the coordinates of the top-left and bottom-right corners of the bounding box of the target object; c represents the confidence score of the target object detection result; and Decoder represents the decoder. <ground>Learnable tokens are inserted into the features of the text instruction to locate words.

[0091] Then, the decoded coordinate values ​​and confidence scores are used to... <ground>By replacing placeholders and using them as input for the next prediction, the output is changed from plain text to text. When a learnable token placeholder is encountered, the token features are extracted, introduced into the decoder network for coordinate decoding, and the placeholder is replaced before proceeding to the next iteration. This is beneficial for large language models to learn high-level semantic features and reduces the illusion caused by positioning output errors.

[0092] To facilitate understanding of the training process of the target generative model, the following explanation will be provided with reference to some specific examples.

[0093] Specifically, model training can be divided into three stages. The first stage is learnable feature transfer localization optimization training, which can adopt the LoRa training method, adding lightweight parameters to train the visual encoder, feature mapping network and large language model, so that the multimodal large language model can adapt to the new form of output. The training task is the localization task.

[0094] In the second stage, since the main objective is to perceive the state of traffic elements in the image, the training text prompts are modified as follows: Text prompt t2: "You are a professional traffic assistant. Based on the vehicle at the specified location, please locate the lane line position {p} of that vehicle in the image, and provide the direction the vehicle should travel. Also, locate the traffic light position {p} and answer the question about the current traffic light status, specifically whether left turns, straight ahead, U-turns, and right turns are prohibited." By constructing these prompts, the model can accurately capture the target state and future trend direction in a single image.

[0095] In the third stage, based on the text prompt t2, multi-image judgment is introduced, and the training prompt is modified as follows: Text prompt t3 = "You are a professional traffic assistant. You will receive multiple images. Please locate the lane line position {p} of the vehicle in the image based on the vehicle at the specified location, and give the direction that the vehicle should travel. At the same time, locate the position {p} of the traffic light. And answer the current traffic light status. Note that you need to answer whether left turn, straight, U-turn, and right turn are prohibited. In addition to answering the basic target attribute judgment, you also need to do the following tasks and make a summary answer: 1. In the first image, determine whether the vehicle is in front of the zebra crossing. 2. In the second image, determine whether the vehicle's travel direction is consistent with the direction determined in the first image and whether it has crossed the zebra crossing. 3. Determine whether the vehicle's travel trajectory conflicts with the prohibited direction indicated by the traffic lights in the first and second images. 4. If you cannot make an accurate judgment, please give a comprehensive judgment based on the remaining images and give a final opinion on whether the vehicle violated the rules by running a red light."

[0096] This prompt integrates target vehicle status judgment, traffic element judgment, multi-image information integration, logical reasoning, and provides a final opinion. By gradually increasing the difficulty of the task, the model can adapt to the judgment conditions of vehicles running red lights in traffic scenarios and provide step-by-step conditional reasoning, which is finally integrated into a complete chain of reasoning evidence to reduce the illusion output of multimodal large language models.

[0097] Figure 6 This is a flowchart illustrating the model training process of a traffic violation identification method according to an embodiment of this application. Figure 6 As shown, in some embodiments of this application, the target generative model can be trained in the following ways:

[0098] Phase 1: LoRa training combined with target localization cue word t.

[0099] LoRA (Low-Rank Adaptation) is a lightweight fine-tuning method used to train an initial generative model for the task of locating traffic elements, particularly for the positional information of vehicles, lane lines, and traffic lights. It adds a low-rank matrix to the weight matrix of the pre-trained model and trains only these newly added learnable tokens. In other words, it adapts to the needs of a specific task by adding lightweight parameters, while keeping most of the parameters of the base model unchanged.

[0100] After receiving text instructions containing specific labeled features (such as learnable tokens), the model generates outputs corresponding to these features. These outputs are then fed into a dedicated decoder (such as a lightweight Transformer decoder) to obtain specific coordinates and confidence information, such as "the lane line is located at (x1, y1, x2, y2), and the traffic light is located at (x1', y1', x2', y2')". By decoding the labeled features generated by the model, the model effectively solves the problem of illusory outputs that may be introduced when the model directly predicts coordinate values ​​token by token, ensuring the accuracy and reliability of the localization results.

[0101] Phase Two: Target Localization and Attribute Perception Training.

[0102] The model is trained by combining target localization and attribute-aware cue words t2. That is, based on the first stage of training, more complex text cue words t2 are used, such as "You are a professional traffic assistant. Please find the position of the lane line where the vehicle is located {p}, give the direction the vehicle should travel, locate the position of the traffic light {p}, and determine whether the traffic light prohibits left turns, straight ahead, U-turns, or right turns." The model needs to output not only the target position, but also the judgment of traffic element attributes, such as the status of the traffic light.

[0103] Phase 3: Comprehensive judgment training based on multi-image information.

[0104] The model is trained by combining target localization, attribute perception, and summary judgment prompts t3. That is, using t3 which contains comprehensive task instructions, the model is required to perform localization, attribute perception, and logical reasoning in multi-image scenarios. For example, "You are a professional traffic assistant and will receive multiple images. Please locate the lane line position {p} of the vehicle, give its direction of travel, locate the position of the traffic light {p}, judge the traffic light status and prohibited direction, and judge whether the vehicle has violated the rules by running a red light based on this information." The model needs to perform continuous multimodal perception and reasoning based on multiple image inputs, and finally draw a conclusion on whether the vehicle has run a red light.

[0105] Through the above three stages of supervised training, the multimodal large language model (target generative model) gradually develops from simple visual target localization to the perception of traffic element attributes, and finally transitions to logical reasoning and illegal behavior judgment based on multi-image information, realizing the upgrade of the model from basic perception to advanced understanding and decision-making.

[0106] To further improve the generalization ability of the target generative model to cope with complex and ever-changing traffic scenarios, the following steps can be performed: Adjust the third text prompt to obtain a fourth text prompt, where the third text prompt guides the second target generative model to analyze traffic elements in the third historical image according to a preset logical order, and the fourth text prompt guides the target generative model to adjust its inference path based on the content of the image to be identified; train the target generative model using the third training dataset and the fourth text prompt to obtain prediction results; determine the reward signal corresponding to the prediction results, where the reward signal guides the parameter update of the target generative model; adjust the model parameters of the target generative model based on the reward signal to obtain the optimized target generative model.

[0107] In some embodiments of this application, the reward signal corresponding to the prediction result can be determined in the following ways: determining a first reward signal corresponding to the prediction result, wherein the first reward signal is used to quantify the degree of matching between the format of the prediction result and the preset format in the fourth text prompt word; determining a second reward signal based on the prediction result and the real annotation corresponding to the third historical image sequence, wherein the second reward signal is used to quantify the degree of difference between the prediction result and the real annotation; and determining the reward signal based on the first reward signal and the second reward signal.

[0108] Specifically, the third text prompt is an instruction guiding the model to analyze traffic elements in the third historical image according to a preset logical order. The fourth text prompt further guides the model to adjust its inference path based on the image content. In other words, the model needs to learn to make flexible judgments based on actual conditions, rather than simply following a fixed logical order. The third text prompt helps the model establish an orderly analysis process, ensuring that the model can systematically identify and locate elements such as vehicles, traffic lights, and lane lines. The fourth text prompt enhances the model's flexibility and adaptability, enabling it to make optimal judgments based on different traffic conditions and image content.

[0109] The first reward signal quantizes the matching degree between the prediction result format and the specified format, while the second reward signal quantizes the difference between the prediction result and the true annotation. The first reward signal ensures that the output follows a preset structured format, facilitating subsequent processing and understanding, while the second reward signal directly evaluates the accuracy of the model's judgment. The combination of the two guides the optimization and adjustment of model parameters to achieve higher recognition accuracy and a more reasonable inference path.

[0110] To facilitate understanding of the reinforcement learning training process described above, the following explanation will be provided in conjunction with some specific examples.

[0111] In some embodiments of this application, a group-based relative reward optimization strategy can be used to train the model. This transforms the approach from strictly adhering to structured output—that is, first locating the target vehicle, then the traffic light, then performing attribute judgment, then summarizing multiple images, and finally performing logical reasoning and summarizing—to allowing the model to make autonomous choices. Unlike general reinforcement learning strategies, this application can also optimize the t3 prompts. The prompts allow the model to first think, locate the effective region in the image based on the thought process, then crop and enlarge the effective region as supplementary data, which, along with historical records, is fed to the model. After multiple rounds of location and enlargement, the model learns rich global and local visual information and finally provides an accurate answer. The steps of a single inference are as follows:

[0112] (1) Question: t4 = "You are a professional traffic assistant. You will receive multiple images. Please locate the lane line position {p} of the vehicle in the image based on the vehicle at the specified location, and give the direction that the vehicle should travel. At the same time, locate the position {p} of the traffic light. And answer the current traffic light status. Note that you need to answer whether left turn, straight, U-turn, and right turn are prohibited. In addition to answering the basic target attribute judgment, you also need to do the following tasks and make a summary answer: 1. In the first image, determine whether the vehicle is in front of the zebra crossing. 2. In the second image, determine whether the vehicle's travel direction is consistent with the direction determined in the first image and crosses the zebra crossing. 3. Determine whether the vehicle's driving trajectory conflicts with the prohibited direction indicated by the traffic lights in the first and second images. 4. If you cannot make an accurate judgment, please give a comprehensive judgment based on the remaining images and give the final opinion on whether the vehicle violated the rules by running a red light. Please analyze and think about each image and locate the effective area. If you want to make a secondary judgment on the details of the target area, please give the image coordinates: "

[0113] (2) Model: <think> Model Thinking Process< / think> <box> [x1,y1,x2,y2]>< / box> ;

[0114] (3) Model: <think> Further consideration based on the location image< / think> <box> [x1,y1,x2,y2]>< / box> ;

[0115] (4) Model: <answer> answer< / answer> .

[0116] During the optimization process, the Group Relative Reward Optimization (GRPO) strategy is utilized for optimization:

[0117]

[0118] in, Let q be the objective function of the multimodal large language model optimized using the GRPO strategy, where q represents the text instruction and v represents the visual input, i.e., images or video frames of the traffic scene. For the action or output generated by the model under the i-th question and the j-th visual input, Let π represent the number of questions and the number of visual inputs for each question, and let π represent the model's behavioral strategy. This represents the probability of behavior under the old model parameter state.

[0119]

[0120] in, The optimized model parameters represent the probability of behavior under each state. The dominance function measures the action. The degree of advantage compared to average performance is used to calculate the relative reward, and ε is a parameter for the cutoff range, used to prevent gradient updates from being too large or too small.

[0121]

[0122] in, For the model to action The reward value is denoted by mean, which represents the average reward value generated across all combinations of questions and visual inputs, and std, which represents the standard deviation of the reward value generated across all combinations of questions and visual inputs.

[0123] The above formula calculates the advantage of the actions generated by the new and old behavioral strategies of the model relative to the baseline expected reward and normalizes them. At the same time, it uses a truncation operation to control the extreme value of the reward ratio and uses a min function to select an appropriate reward for weighted averaging to optimize the model parameters. The relative reward encourages the model to explore better strategies during training, while ensuring the stability of the training process and the generalization ability of the model, and avoiding overfitting to local optima.

[0124] In some embodiments of this application, in terms of reward design, two complementary reward mechanisms are combined in the answer generation process: the first is a format reward: which uses the forced reasoning process... <think> and< / think> The final answer is wrapped in tags. <answer> and< / answer> Tags are used to wrap the output to ensure the standardization and logical coherence of the output structure.

[0125] The second is the accuracy bonus: it adaptively adjusts based on the question type. For location-based questions, the IOU matching score is used as the bonus; the higher the match, the higher the score, and if there is no match at all, the score is 0. In addition, for open-ended questions, the bonus is defined as the word-level alignment between the generated answer and the actual answer, and the average score of three metrics, ROUGE-1, ROUGE-2, and ROUGE-L, is used as the scoring basis.

[0126] Figure 7 This is a schematic diagram illustrating reinforcement learning training for a traffic violation identification method according to an embodiment of this application, as shown below. Figure 7 As shown, in some embodiments of this application, the reinforcement learning training of the target generative model includes:

[0127] Step (1): Obtain the question instruction set t4 and multiple images. The question instruction set t4 is used to guide the model to perform multimodal reasoning. It is similar to "You are a professional traffic assistant. You will receive multiple images. Please analyze whether the vehicle violated the rules by running a red light. Based on what evidence, how do you determine that its behavior is illegal?" At the same time, multiple images containing different traffic scenarios and time points are collected as inputs for model reasoning and learning.

[0128] Step (2): Image cropping and focusing. The model first performs a preliminary interpretation of multiple images and, based on the requirement of "locating the effective area" in the problem instruction set t4, performs image cropping through multiple inferences (for example, the model may first identify vehicles and then try to locate traffic elements such as traffic lights and zebra crossings based on the vehicle positions). After each inference, the model outputs a cropping suggestion (box1 to box4) to crop a local area from the original image. This allows for more detailed analysis of these areas. For example, box1 may point to the vehicle position, and box2 may point to the traffic light area. In this way, the model gradually focuses on the most relevant image information, improving the accuracy of subsequent inferences.

[0129] Step (3): Begin reasoning and obtain multiple reasoning results. The model performs in-depth reasoning based on the cropped image sequence, including but not limited to identifying traffic light status, vehicle direction of travel, and whether the vehicle has crossed the stop line (reasoning 1 to reasoning 8). Each reasoning step is based on currently available information and previous reasoning results. For example, reasoning 1 may determine the vehicle's position, reasoning 2 may assess the traffic light status, and reasoning 3 may determine whether the vehicle has violated traffic rules. This series of reasoning processes is the process by which the model gradually builds a chain of evidence, with the ultimate goal of determining whether there is a traffic violation.

[0130] Step (4): Relative Advantage Calculation. After each inference step, the system evaluates the advantage of the inference action compared to other possible actions. This step involves a policy network that estimates the value or expected probability of success for each inference action. For example, when inferring the vehicle location (Inference 1), if the model chooses to zoom in on the license plate area more helpful for identifying illegal behavior than to focus on the background, the policy network will give this action a higher advantage score. The relative advantage is calculated by comparing the reward of the current action with the reward of the baseline action. The baseline is usually the average level of the model or other preset benchmarks.

[0131] Step (5): Reward calculation for training optimization. Reward calculation is based on two main dimensions: format reward and accuracy reward. Format reward ensures that the model output conforms to a predetermined structured format, for example, using... <think> 、 <answer>The model incorporates inference and the final answer into a label-based system. Accuracy rewards depend on the degree of match between the model's inference results and the ground truth labels, such as the IoU score (for location matching) and the ROUGE score (for text similarity). Combining these two reward signals, the system calculates a total reward value to guide model parameter updates. Through reinforcement learning training, the model gradually learns how to optimize inference paths and strategies to maximize rewards, thereby improving the accuracy and completeness of evidence in identifying traffic violations.

[0132] In some embodiments of this application, the following steps may also be performed: determining a feature vector corresponding to the violation judgment text, wherein the feature vector is used to reflect the semantic content of the violation judgment text; matching the feature vector with legal provisions in the legal database to obtain candidate legal provisions; determining a target legal provision from the legal database based on the feature vector and the candidate legal provisions, wherein the target legal provision includes the legal provision in the legal database that has the highest similarity to the feature vector; and fusing the violation judgment text with the target legal provision to obtain a structured report, wherein the structured report includes the judgment conclusion of the traffic violation of the target object, the factual description of the traffic violation, and the theoretical basis corresponding to the judgment conclusion.

[0133] Specifically, a pre-trained text encoder (such as SigLIP or BERT) can be used to encode the violation judgment text into feature vectors. The encoder converts the semantic information in the text into numerical values, facilitating comparison with provisions in the legal database. By calculating the cosine similarity between the feature vector and the feature vector of each legal provision in the database, the several legal provisions with the highest matching degree are identified as candidate legal provisions. Among the candidate legal provisions, the one with the highest similarity to the feature vector of the violation judgment text is selected as the target legal provision. Finally, key information in the violation judgment text, such as the time, location, and behavior description, is combined with the target legal provision to generate a structured report containing specific information about the violation and its theoretical basis.

[0134] Figure 8 This is a structured report generation flowchart of a traffic violation identification method according to an embodiment of this application, such as... Figure 8 As shown, in some embodiments of this application, a structured report can be generated through the following steps:

[0135] Step (1): Acquire the image to be queried and perform license plate localization and traffic element detection. The system receives real-time or historical image data from the intelligent traffic monitoring system as input (the image to be queried). These images are first localized using advanced target detection technology (such as YOLO), and key traffic elements (such as traffic lights, lane lines, and zebra crossings) are detected and understood using a multimodal perception and understanding module (combining GroundingDINO and SigLIP, etc.). The output results include vehicle position, traffic light status, and the position information of zebra crossings and lane lines. This information is the basis for subsequent illegal behavior judgment and law retrieval.

[0136] Step (2): Input the multimodal large language model and expand the training image library. Input the license plate localization and traffic element detection results obtained in step (1), along with the original images, into the multimodal large language model. At the same time, these results are added to the training image library as new training samples for subsequent model iteration training to improve the model's recognition ability in more diverse scenarios.

[0137] Step (3): Initial retrieval to obtain relevant legal provisions. Based on the output of the multimodal large language model, i.e., the preliminary judgment and description of vehicle behavior, the system uses vector retrieval algorithms such as cosine similarity to perform an initial retrieval in the legal database. The legal database is composed of legal provisions texts that have undergone semantic feature processing (such as using BERT encoding). The goal of the initial retrieval is to locate the legal provisions most relevant to the current violation. These provisions constitute the candidate legal set.

[0138] Step (4): Secondary retrieval and document importance re-ranking. The system compares the output of the multimodal large language model with the legal fragments obtained in the first retrieval, further refines the retrieval through deep learning similarity calculation, and re-ranks the candidate legal documents according to the relevance score. This step ensures that the legal provisions finally cited are highly related to the specific illegal events, which is the theoretical basis for effectively judging illegal behavior.

[0139] Step (5): Integrate the search results with the output of the large language model. The reordered regulatory documents from step (4), along with the original output of the multimodal large language model, are fed back into the large language model as input. This process aims to integrate visual perception information with regulatory knowledge, enabling the model to adjust its original judgment based on more authoritative normative provisions, and form a more accurate and legal conclusion on illegality.

[0140] Step (6): Output a summary document and feed it back to the training library. After integrating the legal information, the large language model outputs a summary document containing elements such as a description of the illegal acts, the basis for judgment, and references to relevant legal provisions. This document is not only a detailed report on traffic violations, but also an important basis for management departments to make decisions. In addition, in order to further improve system performance, the summary document will be fed back to the training image library as sample data for future model training, helping the model learn more diverse illegal behavior patterns and cases, and improving the accuracy and robustness of the system.

[0141] Through the above steps S202 to S208, the goal of accurately identifying and understanding traffic scenarios is achieved, thereby realizing the technical effects of automating and intelligentizing the determination of illegal behavior under complex conditions and significantly reducing the false alarm rate. This solves the technical problem that the identification methods for illegal behaviors such as running red lights used in related technologies rely on single visual detection or rule engines, lack semantic understanding of traffic elements and multimodal collaborative reasoning capabilities, resulting in a high false alarm rate.

[0142] Figure 3 This is a schematic diagram of the overall process of a traffic violation identification method according to an embodiment of this application, as shown below. Figure 3 As shown, it includes:

[0143] Step (1): Obtain target license plate information and video sequence. The system receives a request command containing target license plate information and a continuous video sequence from the intelligent traffic monitoring equipment or data transmission interface. The video sequence consists of multiple image frames, covering the time period from when the vehicle approaches the intersection to when it leaves, providing rich contextual information for subsequent behavior analysis and violation judgment.

[0144] Step (2): Target vehicle localization and tracking. First, vehicle detection: Apply multi-object detection technology (such as YOLOv5 or Faster R-CNN) to identify all potential vehicle targets in each frame of the image and generate bounding boxes for candidate vehicles. Second, license plate magnification and recognition: For each candidate vehicle region, further magnification is performed and a specific license plate recognition algorithm (such as a combination of CNN-based license plate localization and OCR character recognition) is used to extract license plate information for subsequent vehicle matching. Then, nearest neighbor matching: Through continuous frame matching of license plate information, the movement trajectory of the target vehicle in the entire video sequence is tracked to ensure accurate tracking of the target vehicle's position changes from the first frame to the last frame, providing continuous vehicle position data for violation judgment.

[0145] Step (3): Traffic element detection. Using image segmentation technology (such as DeepLab V3+) and a self-attention interactive feature encoder (Transformer), key elements such as lane lines, zebra crossings, and traffic lights are detected and identified in each frame of the video sequence. Multimodal perception and understanding technology ensures that these traffic elements can be accurately identified even under occlusion or complex lighting conditions, laying the foundation for subsequent behavior compliance judgment.

[0146] Step (4): Stage 1 Multimodal Large Model Instruction-Supervised Fine-Tuning. Based on the pre-trained multimodal large model, lightweight fine-tuning techniques such as LoRA are used, combined with customized instruction text (such as "Please locate lane lines" and "Please describe the current traffic light status"), to conduct instruction-supervised training on the model, ensuring that the model can understand and execute specific traffic element location and description tasks. Through supervised training, the model learns how to accurately interpret visual information, laying the foundation for subsequent attribute perception and logical reasoning.

[0147] Step (5): Traffic light status determination. Using a multimodal large language model that has been fine-tuned under instruction supervision, based on specific prompts (such as "Please determine whether the traffic light is red"), the model can understand and predict the current status of the traffic light, including distinguishing between red, yellow, and green lights and the specific direction pointed to by the arrow (such as going straight, turning left, and turning right), providing important clues for the determination of illegal events.

[0148] Step (6): Stage Two: Multimodal Large Model Mind Chain Supervision and Fine-Tuning. A mind chain supervision training mechanism is introduced, using multi-stage reasoning prompts (such as "Please explain why the vehicle still passes through the intersection on a red light") to guide the model in learning how to perform logical reasoning based on the location results of traffic elements to determine whether the vehicle's behavior is legal. This stage particularly emphasizes the model's ability to understand the relationships between different traffic elements and how to coherently integrate these relationships into a complete logical chain, providing support for subsequent comprehensive judgment.

[0149] Step (7): Vehicle driving direction determination. Based on the target vehicle location and tracking, combined with traffic light status judgment and lane line assignment analysis, the model learns how to determine the actual driving direction of the vehicle and compares it with the legal driving direction indicated by the traffic light to determine whether there is a directional conflict.

[0150] Step (8): Stage 3 Multimodal Large Model Reinforcement Learning Inference Fine-tuning. Reinforcement learning technology is used to post-train the model by setting a reward mechanism (such as adjusting the reward based on whether the judgment is correct or not), so that it can autonomously optimize the decision-making strategy in uncertain and complex environments. The model adjusts the parameters based on past experience to improve the accuracy of identifying traffic violations in different scenarios, especially for edge cases that are difficult to be clearly defined by rules, such as the situation of briefly crossing the line to avoid emergency vehicles.

[0151] Step (9): Make a comprehensive judgment and provide a structured chain of evidence. The model integrates the judgment results of all stages, including the real-time location of the target vehicle, traffic light status, vehicle driving direction, lane line attribution, etc., to form a logically rigorous and hierarchically clear chain of evidence. This chain of evidence not only describes the process of the violation, but also includes comprehensive analysis based on multimodal data, providing strong support for subsequent management actions.

[0152] Step (10): Large Language Model Summary. Based on the information of the entire chain of evidence, the large language model generates a detailed text summary, including a description of the illegal event, analysis of key elements, and preliminary judgment, in preparation for the formation of a decision report. This summary process requires the model to present complex information in a human-understandable language form to ensure the readability and persuasiveness of the report.

[0153] Step (11): Summarize decision recommendations and form a decision report. Combining the results of the large language model summary in step (10) and the most relevant legal provisions retrieved from the legal database, the system automatically generates a standardized decision report containing the time, location, description of the illegal facts, theoretical basis, and decision recommendations. This report generation process not only reflects the intelligence of technology but also ensures the compliance of management actions.

[0154] This application proposes a semantic reasoning framework for traffic violations based on a goal-generative model. It introduces a chain-of-thought supervised training mechanism into the red-light violation determination task, guiding the model to progressively reason about "vehicle location → driving direction → lane marking → traffic light prohibited direction → legality of behavior," achieving fine-grained and traceable logical judgments and significantly improving the accuracy of discrimination in complex intersection scenarios. Furthermore, this application proposes a multimodal reinforcement learning fine-tuning strategy for traffic scenarios, enabling the large language model to dynamically integrate visual elements (license plates, traffic lights, zebra crossings, lane markings) with spatiotemporal context. Without explicit rule encoding, it autonomously learns the generalization ability to determine "when, where, and what type of vehicle constitutes a violation," effectively addressing the challenges of transferring judgments across different cities, time periods, and lane marking standards. Furthermore, this application also proposes a "retrieval-enhanced generation (RAG) + structured output of evidence chain" mechanism. After the model makes a determination of illegality, it automatically associates video frame evidence, semantic reasoning paths and legal provisions to generate a structured decision suggestion report with judicial effect. This achieves closed-loop automation from "video perception" to "document generation" for the first time, greatly improving management efficiency and the interpretability of results.

[0155] This application and related technologies all adhere to the premise of locating key elements of red-light running to obtain key basic information from video frames. However, this application optimizes the location of key information by adding a multi-level verification mechanism to ensure the accuracy of license plate and other location information. Specifically, addressing the problems of relying on manual rules, isolated element detection, and lack of semantic reasoning and interpretable output in urban intersection surveillance videos, this application proposes an end-to-end intelligent identification and evidence generation method for red-light running violations based on a target generative model. During model training, a thought chain supervision mechanism is introduced to guide the model to gradually reason about the logical consistency of vehicle trajectory, lane affiliation, light color semantics, and driving direction, achieving fine-grained behavior legality judgment. Simultaneously, a reinforcement learning fine-tuning strategy is adopted to enable the model to dynamically integrate multi-source visual elements and adaptively learn cross-scene violation discrimination capabilities without explicit rule intervention. In the reasoning stage, an innovative retrieval-enhanced generation technology is introduced to automatically associate the model's judgment results with traffic regulations and key frame evidence, generating a structured decision suggestion report, achieving a closed-loop process from perception to decision-making to document output. At the system deployment level, this method supports automated labeling of illegal events and archiving of evidence for massive intersection video streams, without the need for manual formulation of judgment rules or additional labeling samples, significantly improving the level of intelligence in traffic management and the traceability of judicial evidence.

[0156] It should be noted that the application scenarios of this application include, but are not limited to: (1) Intelligent traffic violation automatic evidence collection system: By deploying cameras at intersections, this application can automatically identify pedestrians, non-motorized vehicles or motor vehicles running red lights, generate structured violation evidence (image / video clip + timestamp + location + object category), replace manual review, and improve management efficiency and accuracy. (2) Urban traffic management big data platform: As the core identification engine of the city-level traffic violation database, it supports the automatic labeling and archiving of red light running events for massive intersection videos, assisting traffic management departments in conducting violation trend analysis, hot intersection management and management resource optimization. (3) Public safety and emergency response linkage: In key areas such as large-scale events, school surroundings, and hospital entrances, this method can be used to realize real-time identification and alarm of high-risk red light running behavior, link relevant personnel to respond quickly, and prevent accidents from occurring. (4) Insurance and judicial evidence support system: Provides highly reliable automated video evidence extraction services for traffic accident liability determination and insurance claims, reduces the cost of manual evidence collection, and improves the efficiency of judicial and insurance processes.

[0157] Figure 9 This is a structural diagram of a traffic violation identification device according to an embodiment of this application, such as... Figure 9 As shown, the device includes:

[0158] The acquisition module 902 is used to acquire multiple images to be identified, wherein the multiple images to be identified include a time-continuous image sequence of the target object in a traffic scene;

[0159] The detection module 904 is used to perform target detection on multiple images to be identified and obtain detection results, which include the movement trajectory of the target object and multiple traffic elements in the traffic scene.

[0160] The filling module 906 is used to fill the preset text prompt words with the target object and multiple traffic elements to obtain a prompt instruction set. The prompt instructions in the prompt instruction set are used to guide the target generative model to analyze the traffic elements in the image to be identified according to a preset logical order.

[0161] The determination module 908 is used to process the prompt instruction set and multiple images to be identified using a target generative model to obtain violation judgment text. The target generative model is used to capture the semantic relationship between multiple traffic elements, and the violation judgment text is used to determine the traffic violation behavior of the target object in the traffic scene.

[0162] It should be noted that, Figure 9 The traffic violation identification device shown is used for enforcement. Figure 2 The method for identifying traffic violations shown is therefore Figure 2 The relevant explanations and instructions in the methods for identifying traffic violations also apply to... Figure 9 The traffic violation identification device shown is not described in detail here.

[0163] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.< / answer> < / think> < / ground> < / ground> < / ground>

Claims

1. A method of identifying traffic violations, characterized by, The method comprises the following steps: acquiring a plurality of to-be-recognized images, wherein the plurality of to-be-recognized images comprise a time-continuous image sequence of a target object in a traffic scene; performing target detection on the plurality of to-be-recognized images to obtain a detection result, wherein the detection result comprises a movement track of the target object and a plurality of traffic elements in the traffic scene; filling a preset text prompt word with the target object and the plurality of traffic elements to obtain a prompt instruction set, wherein a prompt instruction in the prompt instruction set is used to guide a target generative model to analyze traffic elements in the to-be-recognized images according to a preset logical sequence; processing the prompt instruction set and the plurality of to-be-recognized images by using the target generative model to obtain a violation judgment text, wherein the target generative model is used to capture semantic associations between the plurality of traffic elements, and the violation judgment text is used to determine a traffic violation behavior of the target object in the traffic scene.

2. The method of claim 1, wherein, The target object comprises a target vehicle in the to-be-recognized images; performing target detection on the plurality of to-be-recognized images to obtain a detection result, comprising: detecting a first to-be-recognized image in the time-continuous image sequence to obtain a plurality of candidate vehicle images corresponding to a plurality of candidate vehicles, wherein the first to-be-recognized image comprises a first frame image in the time-continuous image sequence; performing license plate detection on the plurality of candidate vehicle images respectively to obtain a target vehicle image matched with a preset license plate; determining first visual semantic information corresponding to the target vehicle image; determining a first detection result corresponding to the target vehicle according to the first visual semantic information, wherein the first detection result comprises a driving track of the target vehicle.

3. The method of claim 2, wherein, Determining a first detection result corresponding to the target vehicle according to the first visual semantic information, comprising: determining license plate information and second visual semantic information of a vehicle in a second to-be-recognized image in the time-continuous image sequence, wherein the second to-be-recognized image comprises any one to-be-recognized image in the time-continuous image sequence except the first to-be-recognized image; determining a first similarity score according to the license plate information and the preset license plate, and the first visual semantic information and the second visual semantic information, wherein the first similarity score is used to quantitatively represent a similarity degree of the vehicle and the target vehicle, and a ratio of a similarity weight of the license plate information corresponding to the second to-be-recognized image and the second visual semantic information decreases with an increase of a time sequence; determining a driving track of the target vehicle in the time-continuous image sequence according to the first similarity score to obtain the first detection result.

4. The method of claim 1, wherein, Performing target detection on the plurality of to-be-recognized images to obtain a detection result, comprising: detecting the plurality of to-be-recognized images by using a visual detection model to obtain a plurality of candidate bounding boxes comprising candidate traffic elements; determining a description text of an image region corresponding to each of the candidate bounding boxes, wherein the description text is used to describe a candidate traffic element name in the candidate bounding box; The description text is semantically matched with a preset traffic element name to obtain a second similarity score, and the second similarity score is compared with a preset threshold to obtain a comparison result; A second detection result corresponding to the traffic element is determined according to the comparison result.

5. The method of claim 1, wherein, The target generative model is trained in the following manner: The first training data set and the first text prompt word are used to train an initial target generative model to obtain a first target generative model, wherein the first training data set includes a first historical image with traffic element annotation, and the first text prompt word is used to instruct the initial target generative model to locate the traffic element, and the first target generative model is used to locate the position of a first target object and a first traffic element in the first historical image; The second training data set and the second text prompt word are used to train the first target generative model to obtain a second target generative model, wherein the second training data set includes a second historical image with traffic element annotation and state annotation of the traffic element, and the second text prompt word is used to at least instruct the first target generative model to judge the state of the traffic element, and the second target generative model is used to judge the state and trend of a second target object and a second traffic element in the second historical image; The third training data set and the third text prompt word are used to train the second target generative model to obtain the target generative model, wherein the third training data set includes a third historical image sequence with traffic element annotation, state annotation of the traffic element and target object illegal annotation, and the third text prompt word is used to at least instruct the second target generative model to perform time-continuous reasoning on the target object and the traffic element, and the target generative model is used to determine a traffic violation conclusion corresponding to a third target object in the third historical image sequence.

6. The method of claim 5, wherein, The initial target generative model includes a visual encoder, a feature mapping network and an initial generative model, wherein the visual encoder is used to convert an input image into local image features, and the feature mapping network is used to spatially align the local image features with the initial generative model.

7. The method of claim 6, wherein, The first training data set and the first text prompt word are used to train an initial target generative model to obtain a first target generative model, including: The visual encoder of the initial target generative model is used to determine a feature sequence corresponding to the first historical image, wherein the feature sequence is used to capture local information in the first historical image; The feature mapping network of the initial target generative model is used to fuse the feature sequence with a text instruction feature corresponding to the first text prompt word to obtain a fusion feature vector, wherein the text instruction feature has an embedded marker feature, and the marker feature is used to instruct the initial target generative model to locate the traffic element; The initial generative model of the initial target generative model is used to process the fusion feature vector to obtain a predicted text, wherein the predicted text includes a predicted position of the first traffic element determined according to the label feature; A loss between the predicted position and the traffic element label is determined, and a model parameter of the initial target generative model is adjusted according to the loss to obtain the first target generative model.

8. The method of claim 7, wherein, The initial generative model of the initial target generative model is used to process the fusion feature vector to obtain a predicted text, including: The initial generative model is used to extract a target label feature from a text instruction feature in the fusion feature vector, wherein the target label feature is determined according to a placeholder in the text prompt word; A decoder is used to decode the target label feature to obtain a predicted position corresponding to the target label feature, wherein the predicted position includes a predicted coordinate and a predicted confidence of the first traffic element; The initial generative model is used to replace the placeholder corresponding to the target label feature with the predicted position to obtain the predicted text.

9. The method of claim 5, wherein, The method further includes: The third text prompt word is adjusted to obtain a fourth text prompt word, wherein the third text prompt word is used to guide the second target generative model to analyze traffic elements in the third historical image according to a preset logical sequence, and the fourth text prompt word is used to guide the target generative model to adjust an inference path according to content of an image to be recognized; A third training data set and the fourth text prompt word are used to train the target generative model to obtain a prediction result; A reward signal corresponding to the prediction result is determined, wherein the reward signal is used to guide parameter updating of the target generative model; Model parameters of the target generative model are adjusted according to the reward signal to obtain an optimized target generative model.

10. The method of claim 9, wherein, Determining a reward signal corresponding to the prediction result includes: A first reward signal corresponding to the prediction result is determined, wherein the first reward signal is used to quantitatively represent a matching degree between a format of the prediction result and a preset format in the fourth text prompt word; A second reward signal is determined according to the prediction result and a real label corresponding to the third historical image sequence, wherein the second reward signal is used to quantitatively represent a difference degree between the prediction result and the real label; The reward signal is determined according to the first reward signal and the second reward signal.

11. The method of claim 1, wherein, The method further includes: A feature vector corresponding to the illegal judgment text is determined, wherein the feature vector is used to reflect semantic content of the illegal judgment text; The feature vector is matched with a regulation provision in a regulation database to obtain a candidate regulation provision; A target regulation provision is determined from the regulation database according to the feature vector and the candidate regulation provision, wherein the target regulation provision includes a regulation provision in the regulation database that has the highest similarity with the feature vector; The method further includes: A feature vector corresponding to the illegal judgment text is determined, wherein the feature vector is used to reflect semantic content of the illegal judgment text; The feature vector is matched with a regulation provision in a regulation database to obtain a candidate regulation provision; A target regulation provision is determined from the regulation database according to the feature vector and the candidate regulation provision, wherein the target regulation provision includes a regulation provision in the regulation database that has the highest similarity with the feature vector; The illegal judgment text is fused with the target legal provision to obtain a structured report, wherein the structured report includes a judgment conclusion of a traffic illegal behavior of the target object, a fact description of the traffic illegal behavior, and a theoretical basis corresponding to the judgment conclusion.

12. An apparatus for identifying traffic violations, characterized by Comprise: An acquisition module is configured to acquire a plurality of to-be-identified images, wherein the plurality of to-be-identified images include a time-continuous image sequence of a target object in a traffic scene; A detection module is configured to perform target detection on the plurality of to-be-identified images to obtain a detection result, wherein the detection result includes an action track of the target object and a plurality of traffic elements in the traffic scene; A filling module is configured to fill a preset text prompt word with the target object and the plurality of traffic elements to obtain a prompt instruction set, wherein a prompt instruction in the prompt instruction set is used to guide a target generative model to analyze traffic elements in the to-be-identified images according to a preset logical sequence; A determination module is configured to process the prompt instruction set and the plurality of to-be-identified images by using the target generative model to obtain an illegal judgment text, wherein the target generative model is used to capture semantic associations between the plurality of traffic elements, and the illegal judgment text is used to determine a traffic illegal behavior of the target object in the traffic scene.

13. An electronic device, comprising: Comprise: A memory and a processor, the memory is used to store program instructions; the processor is connected with the memory, and is used to execute the traffic illegal behavior identification method of any one of claims 1 to 11.

14. A non-volatile storage medium, comprising: The non-volatile storage medium includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the traffic illegal behavior identification method of any one of claims 1 to 11 by running the computer program.

15. A computer program product comprising computer instructions, characterized in that, The computer instructions are executed by the processor to implement the traffic illegal behavior identification method of any one of claims 1 to 11.

Images